How we manage to schedule Machine Learning pipelines seamlessly with Airflow and Kubernetes using KubernetesPodOperator

To provide the best user experience, Dailymotion relies a lot on machine learning algorithms and AI which are behind its recommendation and video tagging engines. So being able to smoothly build, train and deploy new models is one of our main concerns as data engineers. Let’s see how Apache Airflow and Google Kubernetes Engine play so well together and allow us to achieve that goal.

The complex life cycle of a machine learning model

To get a sense of the challenges to bring a model from exploration to production, let’s have a small reminder of the different steps in the life cycle of a machine learning model.

As shown in the above chart, we have different people involved, data scientists and data engineers as well as different environments having their own requirements and purposes. …