Notes on managing a data science team

this is a letter to the self, a loaded lasagna of offhanded notes on what works when managing a data science team.

(an unneccesary) Photo by ALLAN LAINEZ on Unsplash

hi there,

• build a great team. you cannot build the golden gate bridge with insurance bankers and tea sellers. the right tools for the job make it possible. start with building a great team.
• control one variable. management is not always easy. you will be head over heels figuring out the solution to a convoluted problem, and people around you will be aware of bits and pieces of the said problem. when all else seems haywire, look for the single variable that you can control, and leverage upon it. say someone asked you to sell a million teas next to the golden gate bridge. do not immediately jump on refuting the viability of the proposition. ask yourself, and your bossman,

how many days do I have to sell a million teas?

(one million? — easily doable.)

one week, says the boss.

ok. do I decide the audience?

(selling a million tea bags to a Tesla exec is still doable. will probably do them more good than harm, destress the folks building AI that will run giant metal tins on our roads and guarantee our safety*.)

in person, bossman replies.

ok. do I need to make a profit?

…

you get the point. CONTROL at least one VARIABLE. the point of this exercise is to get you to understand your cards, you can always optimize the leveraged variable at the end. the variable could be anything, lean on your morals when in doubt. lean just enough as to not shatter them.

• a happy team is a productive team. no one wants to voluntarily do a 9–5 job in a suffocating environment.
• do not get lost in optimizing for the short term at the cost of the long(er) term. a sure shot way to wind yourself down into a large well of unnecessary problems.
• management is politics.
• management is playing the therapist. do not suck at listening to people in your team, else they will never open up about the actual problems.
• it is 2023 and people still don’t understand data science (read AI). expectations from a data science team would range from simple engineering tasks that have nothing to do with data or science; to expecting near-perfect — nah, perfect solutions amazingly complex problems. whip up the old dough and build me AGI. know and understand what your team is meant to build, and ever so gently steer focus in that general direction.
• brand, advertise, educate. can we slap chatGPT here and make it work like this? the only way the rest of the (engineering? product? business?[that’s a stretch]) org gets up to speed on what a data science team can and cannot do is via learning. if you cannot explain something, you don’t understand it well enough. simple as that. no exceptions. hold knowledge sharing sessions, lead them.

• you hire professionals for a reason. listen to them. flesh out arguments instead of going with a narrow focus and a predetermined outcome. ask provoking questions that cover the essential ground around any given problem, only if they don’t come up organically. (and over time, they will). if you did the first point correctly, you should have the brains necessary to make a good product. your role as a manager would be to clear their way for work.
• something I learned from my old manager — be the last one to speak. in meetings and discussions, in 1o1s and in team rituals. in my interpretation, let people run things, as a team. autonomy is good. authoritarian regimes are (mostly) obsolete for a reason.

• probably arguable but, organize. at the point in my life when I wrote this, I preferred a highly organized way of working within the team. (the very opposite of how this letter is organized.)
  in practical terms, this meant team rituals (stand-ups, plannings, retrospectives, refinements, etc) were nailed down, and not moved around a lot. the team gets used to this and they can better focus on their work. over time, each day of the week will have an associated significance, something like Wednesdays are good for the team to interact with stakeholders, Fridays are a no-go for prod deployments.

I started this point with probably arguable because there might be a better way of doing this, again, talk with your team.

my calendar looked so clean and organized I could have it printed on a t-shirt.

months prior, my calendar looked so ugly and mismanaged, I could have printed it on a t-shirt.

• retrospection is the worst kind of spection. and yet, it is absolutely necessary. I read this somewhere, might as well put it down — there are three kinds of people, ones who learn from their own mistakes, ones who learn from the mistakes of others, and the ones that never learn.

now, learning from others’ mistakes is better than making your own mistakes and learning from them BUT not learning from your own mistakes is worse that not learning at all.

read this twice.

• I say retrospection is the worst kind of spection because I simultaneously believe that you can never have enough data at any given point to make a completely failure proof decision. no matter how trivial a decision can seem. put a figurative time box around coming to a decision, and do your best to have all the necessary data in. and go ahead and take the decision. another note that I make here is, do not sit and warm the eggs of failure, yes, you will make wrong decisions, assess the situation, do the necessary reparation and move on. in action points — post-mortem reports on incidents (YES), quarterly retrospectives and reviews (YES), working on the action points in these reports (BIG YES).

• change courses like water my friend. but don’t work in a waterfall way. print out t-shirts saying “agile or die” and make it your philosophy. change your north star every other full-moon day.
• or don’t. see what works. agile is a good approach for its reasons. waterfall is a good approach for its own. compare, evaluate, settle, repeat at a later point. this point is generic, does not just apply to the working of the team but also your own day-to-day work.

• employees are grown-ups. let them handle the news as it is. there is no need to sugarcoat shit (read: layoffs), have fake promises (read: promotion cycles). be an ally to your team, and the rest will take care of itself. employees are adults who have their own ways of processing information. this stirs the pot of hire-fucking-great-people-in-the-first-place once again.

• maybe some words about interviews — interviews are a great place to look into the life of a fellow professional. a fellow human being. before going into an interview, have a mental model of the culture fit that your team needs, have a written model of the technical skills your team needs. do yourself the courtesy of understanding as much about the person from the CV as possible to dive deeper into the topics that interest you. do not look at interviewing as a task, it is an opportunity. as corny as that might sound. learn something new, take notes.

maybe some words about data science —

• a great author once said, it is 2023 and people still don’t understand data science (read AI). the author believes that the job of a pure engineering team is to build something. to engineer something. the job of a data science team is to figure out how to solve a problem. the scope of a data science team is not restricted to pure data by any authority. if the task at hand is to build a forecasting model for prices of tea at a tea stall next to the golden gate bridge, no one is going to kill you for asking the owner of the tea stall directly. think beyond the box. coming back to the real world where problems are much more nuanced and people much more asocial, figure out simple solutions. work your way up from there. not every machine learning model has to have an NVIDIA H100 doing the labor. do not bring an axe to chop some cilantro, they say.
• keep your team abreast of something new in the field of data science. as a former data scientist, nothing kept me up at night as did trying to weasel out a few accuracy improvements from an existing model using something new. trying to beat control. and most of the stuff that I tried in these waking hours of the night came from an idea. an idea that had its seeds in a place where the collective(read: data scientists) discussed something specific at length. a chapter (or a guild, or another fancy word for a regular meeting to discuss a topic) works wonders to promote knowledge sharing. set this up.
• a very high part of your work will be reiterating on what’s already there, making slight improvements here and there, laying the foundation for something great that will be built (a long long time) down the line, and proving that data science does, in fact, add value.
• do not miss the point of data science. for me, it is to have the best possible backing to your arguments. the best-possible part is made possible by harnessing data, looking at the obvious and not-so-obvious insights, using them to form hypothesis, testing them out, and ultimately making a (statistically significant) difference. the rest is history.

• if you wish to transition from being a data scientist to a data science manager, outside of checking all the obvious checkboxes, make sure to grow a thick skin. middle managers are squeezed from both ends.
• if done right, management can be extremely rewarding. the results are tangible and the feedback loops can be as tight (or loose) as you prefer.

regards,

your past self.