Why is Scrum a terrible idea for machine learning

Scrum, a definition

Let’s start with the definition of Scrum and its peculiarities. Scrum is an agile framework for developing products by breaking down them in small chunks that can be time boxed from 2 to 4 weeks. Such time boxes are called sprints. Progress is tracked in meetings called daily scrums. From this definition it looks like something that seems to work, right? Wrong.

Scrum makes sense in very complex environments in which the product has been defined in its entirety. The time box nature of sprints helps to identify blockers and dealing with requirements or market changes. In such a context, Scrum is merely an execution tool, not a philosophy.

The most common scenarios

Let’s see what happens in the majority of scenarios, in which the product is not entirely defined and needs to be discussed with stakeholders to constantly be arranged. In such cases, sprints create the urge of finalizing a piece of the product in a time box. The intrinsic problems of such a goal are the impossibility of defining 1) the size of said chunks and 2) the duration of the sprints as well. The success of the entire process would depend on two of the most critical variables of project management. In 100% of the cases, I have seen estimating such variables simply by flipping a coin. Another important issue I see with Scrum is the fact that it requires the entire team, not only to be disciplined but also to organize and prioritize their work beforehand. That is yet another critical variable usually estimated by watching the stars.

There’s more. Stakeholders are not supposed to participate in sprints. I have seen entire teams failing miserably when they have had the brilliant idea of bringing stakeholders in their conversations. Stakeholders usually ended up derailing objectives to what their feeling about the product was in that very moment or based on their understanding (which was usually very different from what engineers thought).

How does all this play with data science?

The realm of data science represents one of the most dynamic/empirical environments to build products. The biggest misconception of the scrum in data science projects is that it can set a structure and track it. Which is just like treating the adult lion from the jungle as the pet on the couch (and pretending the lion will listen). To start with, agile is a terrible idea for data science, and Scrum is a terrible execution of a terrible idea.

The nature of data science is to defining problems together with stakeholders and solve them in a creative — rather than hyper structured — way. This is a key differentiator with how engineering projects typically work. The number of unknowns in data science usually stays high even when the machine learning algorithm is robust enough to go to production. Applying Scrum to a data science project is the equivalent of disassembling a car and pretending to reassemble it as a plane. As a result, Scrum in data science typically ends up micromanaging scientists and pissing them off (or worst, making them feel frustrated). Engineers and data scientists are not the same breeds and they should not receive the same treatment.

From a more practical perspective, a data scientist who is changing her priority because data are not available or sufficient for the use case, quality of data is too low concerning the initial thoughts, etc. is quite normal. A data engineer who is changing her priority is probably thinking of changing her job. Both scenarios would make a Sprint goal completely useless. But the former is orders of magnitude more common than the second one.

Ok, Scrum sucks. So what?

Probably the best way to manage and structure data science projects in not-so-complex teams is using the kanban. My personal preference of course is Trello (though many others do the job). The typical setup I use is based on four lists:

Setting up a deadline on cards that stay In progress is the equivalent of the Sprint duration, though much less strict and more flexible. As a matter of fact, new insights from data can change many priorities and trash the entire Sprint. In contrast, with kanban, new priorities would affect only a limited set of cards.

Takeaway

Don’t try to time box tasks that do not depend on time.