From Proof of Concept to Production with data science

Putting data science models into operation and letting them create the promised value.

More and more organizations — small and large — are coming out of the Proof of Concept (PoC) phase of developing data science models, varying from classical machine learning techniques to deep learning or advanced statistics. They’re now up to the task of putting these data science models into operation and let them create the promised value. We call this production.

To be able to get data science models to work and keep working, organizations need extensive IT capacity and expertise next to their data science team. This IT capacity is often called DevOps. DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). Sometimes DevOps applied to machine learning is called “MLOps”. A practice where you combine machine learning development with machine learning operations.

Many organizations, therefore, do rely on both data scientists and DevOps specialists to make their data science magic work. And that’s easier said than done.

We speak different languages

Data Scientists like to experiment. Their purpose and motivation lie in experimenting on vast amounts of data to understand it, find patterns and build data-driven models from it. Their expertise is a mixture of skills, like mathematics, basic IT skills (like Python programming), and some domain knowledge.

DevOps engineers do not like to experiment. Their task is to create robust and trustworthy software operations, on which the business relies. They need to make sure that all software made is understandable, controllable, and reproducible. When something happens or goes wrong, they should know first.

With data science moving to production, it often leads to DevOps engineers touching in the dark. Let’s illustrate this.

Maria & John

Maria is 27 years old with a background in math and starts working for company X. Maria knows many theoretical things about data science and has some practical skills developed as a data scientist in the startup she previously worked for. Her favorite topic is deep learning, and she’d love to start experimenting with the big data sets that company X provides. After several months, with some blood, sweat, and tears, Maria created her first deep neural net, which she trained on a local server provided by the company. She can’t wait to see its first results in real life.

Maria now wants to test her model in a more real-life setting, also referred as staging. For this Maria will need the help of system architect John, who is working in the IT department of company X and has experience with DevOps. Her question to John is straightforward. She wants him to put the deep neural net into production, by connecting it to a periodic stream of data and finally store the results in a separate database. Ideally, it would run for two months. Her model is written in Python.

John is 43 years old, and works almost for 10 years in company X. He likes predictability, and his favorite languages are C++ and Pascal. John likes prototyping, but only in his spare time with his Raspberry Pi. He makes sure the DevOps operations at company X are running smoothly. Because the company’s data is highly sensitive, management decided to keep all data on-premises, stored in local servers.

When entering the IT department Wednesday morning, Maria confronts John with her question. After hearing her story, John asks Maria: “Are you sure that the model is stable, reliable and production-ready?”. Maria answers that she doesn’t know, because it’s an experiment. She wants to find out how the model is performing in a more real-life setting. John says that he normally needs approval from management to spend DevOps resources on projects that are proven to be production-ready and are proven to create value to the company.

John feels for Maria and says that he sees what he can do for her. However, John needs some other answers. “Can you tell me how much memory and computing power the model needs?” Maria tells him it will consume 100GB of data per week. John doesn’t like the answer. “Which language and software packages did you use?”. Maria answers that she’s using Python, Scikit Learn, Pytorch and Jupyter Notebooks. John nods silently. Maria starts to become nervous and tells John, “Well I looked at how I could help, and looked on the internet, do you have experience with Docker containers and Kubernetes?” John’s eyes start gazing.

It’s precisely this situation, and probably many others, that companies have to face that the vast majority of data science models never reach production.

That’s difficult to cope with, given the potential of technologies like machine learning, but also the amount of money we spend on it.

But it doesn’t stop there. Once a model is in production, many new questions will arise. Almost certainly, once the first version of the model is put to production, it will not perform as expected. A second version is needed. Maria needs to come into play again and she needs to deliver a new version of her model. John, needs to take over again to roll out the new version accordingly. The whole process starts over again.

Continuous iterations of improvements and optimizations are needed to get to the best result. That is how data science works. Often many versions of the same type of model are created and many handovers exist between the data scientist and the DevOps engineer. This is what we call the data science life cycle.

During this lifecycle, many questions between the data scientist and DevOps engineer will rise.

· What is the latest version of the model (version control)?

· On which specific data set was this model trained (metadata)?

· What was the specific output of this model (metrics & logging)?

· Does it perform better than the previous one (A/B testing)?

· Can we easily monitor and benchmark the performance of our models (performance monitoring)?

· If it’s not working well enough, can we easily roll back to the previous one (rollbacks)?

· Who is responsible for the latest model version, and who has the right authorization to make a change in the production process (Users & Permissions)?

I hope at this point you start to understand that it can be quite the challenge to productize a data science model, no matter how simple it is.

How to deal with this?

It boils down to how data scientists and DevOps engineers can start working together effectively with right the mindset and the right tools. Firstly, they need to understand and accept that they both speak different languages but will need one another. Secondly, they need to accept that they will probably need some tools to make their life easier. That means that the data scientist still can understand how she can deploy her models, without needing DevOps skills. While the DevOps engineer on the other hand understands what the data scientist does, and is able to manage and monitor her models and pipelines accordingly in the production environment.

In the next topic, I’ll explain what type of functionalities such tools would need to fit the requirements of DevOps and Data Scientists.

Do you want to stay up to date with similar topics? Follow our blog on Medium or website.