Data Scientists: The Challenge of Managing Stubbornly Autonomous Experts

Big Data Evangelist, IBM

Data science can only function as a sustainable business resource if it’s managed professionally. Regardless of how your organization chooses to organize your data scientists, you need a layer of professional management. The core reasons for this are clear:

  • keep data science initiatives aligned with business requirements,
  • avoid unnecessary overlap among disparate projects,
  • encourage development and sharing of internal best practices, and
  • foster cross-disciplinary collaboration.

One approach for managing data scientists is to establish internal centers of excellence, per my blog from more than a year ago. As I stated there, it could be a formal or informal program in your organization, but it rarely springs up spontaneously, and it’s extremely unlikely to sustain itself without some level of management oversight. To the extent that it comes into being, a center of excellence will usually be the pet project of one or more practicing data scientists who seek to bring more professionalism to their organization’s stewardship of this skillset.

Are data scientists prepared to take on greater management responsibilities? They risk dead-ending their careers if they fall victim to the so-called “Peter Principle,” which Wikipedia defines as “employees tend to be given increasing responsibility and authority until they cannot continue to work competently.” Even if data scientists prove to be competent managers, they may confront the usual jealousies, turf wars, private agendas, dueling personalities and other issues that beset anyone trying to direct the energies of a complex organization.

And, of course, some data scientists may prove difficult to manage under any circumstance, regarding themselves as skilled artisans, “rock stars,” or what have you. Data scientists are often a stubborn, proud and independent group of professionals. They’re paid to be experts and may need to constantly demonstrate that fact out loud in front of the whole team on a regular basis. If they’re especially good at what they do, prima donnas may be given free rein to do what they want. If they’re sly, they may give lip service to your administrative guidance while doing everything in their power to avoid putting it into practice.

Yes, there are people with attitude problems in every profession. This is not to say that most data scientists aren’t well-adjusted individuals who know how to compromise. But you need to be a respected data scientist, first and foremost, in order to succeed as a manager in such a professional culture. And you need to be tactful. It can be difficult to manage anyone who pushes back at everything you say and do, who questions your intelligence and competency, and may have little respect for anybody in management.

That’s the takeaway from this recent article, in which Istvan Hajnal shares his own experiences as a manager of data scientists:

“I noticed...that data scientists, but also statisticians and some top coders, often have difficulties in accepting orders from managers who don’t have technical skills themselves. This does not mean that they would publicly disobey, but rather they would use some technical excuse to do whatever they wanted to do, knowing very well that the manager didn’t have the technical knowledge to challenge them. Coming from an IT and statistics background gave me (just enough) credibility to be taken seriously, and that gave me a head start compared to other managers.”

Rather than just gripe about data scientists being unmanageable so-and-sos, Hajnal describes a brilliant approach for responding to the incidents of non-compliance (deliberate or otherwise) with management guidance. He organizes these as case-based logic:

  • If  “you follow rules, guidelines, standards, structured processes and so on,” and  “you have no problems,” then  “I applaud!”
  • If  “you follow rules, guidelines, standards, structured processes and so on,” and  you experience “you have problems,” then  “I help.”
  • If  “you don’t follow rules, guidelines, standards, structured processes and so on,” and  “you have no problems,” then  “I tolerate.”
  • If  “you don’t follow rules, guidelines, standards, structured processes and so on,” and  “you have problems,” then  “I refuse!” [to tolerate or help].

This approach could apply equally well in any professional organization, not just those involving data scientists. It recognizes that management exists for two excellent reasons: to ensure that the team adheres to the organization’s policies, procedures and controls, and to help the team be as productive as they can possibly be.

Any team member who doesn’t respect the former imperative should not be surprised if management doesn’t bend over backwards to help them out of a jam.

Let's talk more about data scientists

Join me December 4 for a Twitter chat titled "Is Data Science Your Next Career?"

You might also find these posts interesting:

Data Scientists and Their Curriculum - Podcast

Debunking Myths about Data Scientists - Presentation

Or visit the Data Scientist topic page for even more blog posts, videos, podcasts and more