Defining ‘Done’ in Data Science
Observations on Data Science development cycles
Over the past few months the SSENSE Data Science team has been experimenting with using an Agile framework, and I am here to report the interim results. There is an interesting challenge concerning Data Science projects, especially when working alongside developers hoping to use your results at scale. It means handing portions of ongoing research over to those working at faster work cycles than yourself. Data Science and other elements of tech have a lot in common, however our work cycles move at different rates.
One of the key differences between developers and data scientists is defining the completion of a task. A core element of the Agile work environment is the sprint, wherein a set of jobs is accomplished in a brief period of time. Sprints usually span 1 to 2 weeks and contain tasks of varying complexity measured in points (1, 3, 5, 8, …using the Fibonacci sequence). Typically, one point equals a day of work per person and once a task is done, that task’s points are subtracted from the total. These byte-sized (pun intended) pieces of code are the key elements of what a team can be expected to accomplish within the prescribed time frame. Ideally the remainder reaches zero just in time to complete the sprint, though this is not always the case. A key underlying assumption of this point system is the notion of a task being ‘done’. Of course this works best when the jobs are at their most granular.
One of the major challenges faced by the Data Science field in adopting this sprint model is that we generally face fewer, and longer interval questions. These tasks are research-based, hence the future success, and notion of ‘done-ness’, is initially unknown. To explore the related question of deadline estimations, read my colleague Gregory Belhumeur’s piece “Can This be Done by Thanksgiving?”. Eagerness to file away tasks comes with its own set of consequences and there are plenty of examples of long-term questions with ill-defined time limits. Some hypothetical examples include:
- Can we better predict customer conversion rates as a function of past shopping behaviour (compared to current models)?
- Can we recommend more relevant products to our customers?
- More abstractly, can we invent new metrics of success for either case above?
Assuming the answer is ‘yes’ in any case, will we be ‘done’ when we reach our Holy Grail and find an improved model? And if so, is it scalable? Notably, this was the case after the famous Netflix challenge, where the million-dollar winning stacked algorithm was not fully implemented in their recommendation system due to its sheer complexity. Had Netflix demanded more of a speed versus accuracy tradeoff, everyone’s notion of ‘done’ could have dramatically changed. Complex models lead to a related question we are often faced with: Will the new model also provide new insights? I will comment on this in the final project phase, as described below.
Project Phases
Like with most things, there are three distinct phases in a typical Data Science project: a beginning, a middle, and an end. Data Science is fairly comfortable in the first two stages, but deciding where to cut the line at the end is where things become hazy, hence the topic of this piece. I will try to explain it using the diagram below.
The X-axis (time) is loosely defined and can be anywhere from a few days to a few months. The Y-axis (added value of the research to the company) is more abstract, but depending on your ‘metric of success’, has the potential to be properly quantified as added revenue, more website hits, etc. Here are two examples of experiment types in an industry setting:
- Spike Analysis: This is carried out over a few days or weeks. Intended for short-term feedback and testing questions like ‘is X possible?’ or “is it true that Y drives most of our search engine marketing traffic?” The researcher perhaps returns with a model held together with glue, barely working, but he or she gets the basic question answered.
- Production Slant: Having a clear end-goal in mind, the analysis and modelling are designed around delivering production work. In the most extreme case, consider the 1940s Manhattan Project, which began with scant evidence that nuclear explosives were possible, to large-scale Uranium-235 enrichment and its subsequent detonation five years later.
With either kind of project in mind, here are the three phases of a Data Scientists’ contributions to such projects.
The Beginning
The early throes of a project can be meandering, but an enjoyable exercise nonetheless. As a data scientist you should be comfortable venturing into the unknown and performing some basic research of the problem at hand. To quote the great chemist Frank Westheimer:
“Why spend a day in the library when you can learn the same thing by working in the laboratory for a month?”
The payoff is uncertain, but if properly timed, the exploration phase can be rewarding and possibly a time saver. Even a well-documented but failed experiment can be useful for future analysis. This early research phase turns a core tenet of the Agile manifesto on its head, working software over documentation. To wit, a data scientist often needs to document their early work more than they need working code.
The Middle
This is where things get interesting and more visibly busy. The groundwork has been laid, and brainstorming sessions become a reality. Early models are discarded, replaced, or upgraded. Bugs are fixed and prototypes are made to work with more functionality and power. Theoretical problems still lie at the heart of the matter. Ongoing models are questioned and feedback from stakeholders can completely realign priorities, and even upend the question itself.
The End
Finally, we reach the final stretch of a project, and conduct a post-mortem or retrospective analysis. Details are worked out, such as further documentation, debugging, and finishing contracts. We either pass our project on to a team of developers, or the original stakeholders are satisfied with spike summary analysis and the question is laid to rest. Should we stop working on the problem entirely and switch to a new ramp-up phase? Further work on the same problem yields diminishing returns, and time could be better spent elsewhere.
The End…?
But before going too far, however, there is an outcome that we must keep in mind: the breakthrough. While working towards the end of a project, a new jolt of insight could fuel a new phase of productivity unforeseen in the original scope.
Using nuclear explosives again as an example, the invention of the Hydrogen bomb, over 100 times more powerful than its fission counterpart, was only realized as both theoretically and technically feasible in the waning days of the Manhattan project. This burst of additional research, which could have come up empty-handed, fuelled a second R&D phase lasting another decade (admittedly this is a great example in the abstract sense, but a lousy one morally speaking). The real point here is that we do not know if and when an extended project research timeline will result in breakthroughs or end up wasting precious research time.
Concluding thoughts
What to do when faced with the unknown? Although money can be considered wasted in Scenario A, the value-added to the company’s overall strength rarely diminishes. Hence progress is not retrograde; the trade-off is only in the opportunity cost of losing time for another worthy project. To quote Jeff Bezos’ 2018 letter to shareholders:
“Amazon will […] occasionally have multibillion-dollar failures […] We will work hard to make […] good bets, but not all good bets will ultimately pay out. […] The good news for shareowners is that a single big winning bet can more than cover the cost of many losers.”
The simplest advice I can pass along is to balance exploring the benefit of potential breakthroughs in current projects with the eventual gains if time is invested elsewhere. Remember that even before this point is reached, one must allocate a portion of time to open-ended research to know whether such trade-offs even exist. It is cliché for a researcher to say “further research is needed”, but I would advocate for considering future directions if and when pushing the boundaries at the end of a project. In the context of the Agile framework, this means taking a deep, long look at future project directions during retrospective meetings.
Scientists know how to start something, but the real challenge is knowing where to end it. The Agile sprint may not be perfect for Data Science, but it is important to understand it, and how to interact with it. The notion of ‘done’ is a tricky one for scientists. Sometimes a manager will say it is time to cut the cord, and they will be right! But let there be a compromise. Consider taking the side bet, and pursuing small chances if they can lead to big breakthroughs. You don’t need to be right often to make a considerable difference.
Editorial reviews by Catherine Heim, Liela Touré & Prateek Sanyal
Want to work with us? Click here to see all open positions at SSENSE!
