40 Followers
·
Follow

How we manage to schedule Machine Learning pipelines seamlessly with Airflow and Kubernetes using KubernetesPodOperator

Image for post
Image for post
A journey to build Machine Learning pipelines with Airflow and Kubernetes

To provide the best user experience, Dailymotion relies a lot on machine learning algorithms and AI which are behind its recommendation and video tagging engines. So being able to smoothly build, train and deploy new models is one of our main concerns as data engineers. Let’s see how Apache Airflow and Google Kubernetes Engine play so well together and allow us to achieve that goal.

The complex life cycle of a machine learning model

To get a sense of the challenges to bring a model from exploration to production, let’s have a small reminder of the different steps in the life cycle of a machine learning model.

Image for post
Image for post
Life cycle of a machine learning model

As shown in the above chart, we have different people involved, data scientists and data engineers as well as different environments having their own requirements and purposes. …

About

Quach Eric

Data Engineer @Dailymotion

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store