Deploy the stupid model!
Don’t be embarrassed. If your model, however simple, provides some direct and immediate value to the business, deploy it!
For many data scientists, whether seasoned or recently transferred from academia to industry, it can be tough to pitch, develop, and deploy a model that seems trivial, that you know has some problems, and which can certainly be better. However, if a model is going to provide some immediate and positive benefit, especially if low risk, I suggest that you deploy and reap the immediate benefits.
We’ve all been there. You’re starting a project, just getting used to the dataset and thinking though the problem and you create the simplest model you can think of, just to see how it works. I call this the “stupid model.” Perhaps it is a minimal regression model with just a couple of obvious features. Perhaps it is a time series model in which you predict that today’s value is the same as yesterday’s value. Whatever it is, it is dumb; it is unsophisticated; it is shameful. However, if this model has some utility, some incremental value to the business, you should push to prod.
Software development often adopts an agile approach: develop and release the smallest and simplest feature set that provides value, that improves the business. However, data scientists are often reluctant to adopt this same perspective. We have pride and high standards. We would rather hold back, work on it some more, and then present a superior model. I suggest that data scientist should be agile — with caveats of course.
There are few reasons behind my assertion:
Every little bit helps
If the business is asking for your help, they are probably doing so because there is huge potential value and/or they feel that are flying in the dark. Their current process might be the equivalent of a finger in the air.
If the model makes better predictions or recommendations than their current process, it has value.
If there is some simple model that validates their current decisions, it likely has value. Moreover, oftentimes the business will find value in a purely directional output: yes, the model agrees that we should increase (versus decrease) spending on our Facebook channel; No, the model does not recommend that we increase our monthly stock replenishment.
If the business has an existing stupid model but it is fed with an error prone, tedious, and manual process, automation itself can provide value.
It becomes real
As my former boss, Lon Binder, says “if it isn’t in prod, it doesn’t exist.” Once deployed, this becomes real and visible. You now have plenty of motivation to improve the model. This is where you should leverage your pride and standards.
Once deployed, this isn’t some broken promise that the business might have come to expect. You delivered.
The effort is in the data step
Most time and effort in a modeling project is not modeling but obtaining a satisfactory training data set — you’ve written Python scripts to call an API or complex SQL queries to query the databases, or pandas code to transform, clean, and store the data. To get to a stupid model, you did most of the work already. Clean up the queries and scripts, add a process to kick them off, monitor the flows, and you are good to go.
The start of a beautiful relationship
Simple is likely to be understood and more importantly, trusted, by the business. As I’ve written about before, a simple interpretable model, can be the basis of a strong collaboration with the business. Having them onboard with a stupid model provides the foundation upon which to rack up the complexity to more sophisticated and more abstract models that might be more performant but harder to explain.
What if I wait?
Many data scientists view model performance in the following way:
where the area under the curve (here, bars) is the potential total cumulative value, that is, the return to the business, once a model is deployed. Early versions (v0.8, v0.9) are too poor to deploy. You think: “let’s develop the model further, hide away and try some more features, try some other approaches to see whether we can improve RMSE, F1 score, or even training speed. When we are happy with it, proud to share the details, then let’s deploy” (v1.0):
There are two issues here:
1) if the value of early models really is positive, you maximize return to the business by deploying early:
2) businesses are not static. Many are highly dynamic. You don’t always know how long the problem — with current assumptions — you are tackling will persist. Business needs and priorities change. What is relevant today is yesterday’s news:
When should I wait?
So far, I have neglected three major caveats:
1) If deployment adds a lot of additional complexity or maintenance burden for small incremental value, it might not be worth it. You do have to consider the complete sets of costs and benefits.
2) What are the prediction error costs? Obviously, you don’t want to deploy a risky, low performing model for stock buys, cancer diagnoses, or credit card applications. What are the type I and type II error rates and what are the risks and costs? Those have to be factored in too.
3) if the business is not a full partner in this process, rolling out a shoddy model can decrease trust and willingness to adopt and use your data products. Get them onboard early, bought in to agile, and vested in the approach.
To conclude, what I’m trying to get at is not necessarily the idea of “perfection is the enemy of the good,” although it can be, but more but more of Box’s aphorism: “All models are wrong but some are useful.” Maximize useful.
