Why we built and Open Sourced our MLOps python framework

My team and I have open-sourced our python MLOps framework last year. Its name is modelkit and is meant to make ML models reusable, robust, performant and easy to deploy:

https://github.com/Cornerstone-OnDemand/modelkit

Today, I want to share the background and history of its creation and the choice we made.

Our team context

We spent years working on MLOps without any proper framework before properly building our own.

• We are a mature data team: 10 experienced people working on ML models, with all skills necessary to deploy them into production
• We work on complex deep tech NLP: we have around 200 ML models and algorithms currently deployed. Those 200 are interconnected (1 model can call 10 others during its inference).
• We only partially use standardised ML libraries (sklearn, tensorflow, etc.) as most of our logic is custom python code or python wrapped.
• We are production and scalability focused: we have 10millions+ calls a day and frequent batched processing parallelised with hundreds of workers.

The key philosophies that impacted our project:

• We believe that production inference should be separate from the training process. Although this is not true for every team, we don’t need online training or frequent model updates. We like to carefully and manually select the algorithms we want to push to production and optimize them. We therefore believe a model update needs a git change and a service redeploy.
• We believe that model logic should be separate from ETLs processing (Airflow, Spark, …). Model inference should be stateless and one processing is never supposed to take longer than a second.
• We hardly use notebooks. Although they are useful in data exploration, most team members do not use them. In particular, no code is released to production without a peer review in git.
• We want all configurations and model development processes to be on git, not in a UI, not in notebooks, and not in a framework specific database. MLOps should follow the “Configuration as Code” logic and not the "no code" one.

Where we felt we needed to tool ourself

We built a python framework focused on "model inference". Its key goals are:

1. to structure how we write our python code for all interdependent ML models and algorithm
2. to make it robust and easy to optimise
3. to make it easy to deploy in any production environment
4. to make it easily sharable between team members or teams

Why most solution/framework on the market didn't fit our needs:

• Most ML framework focus on non-mature teams, especially concerning deployment skills (the usual "jupyter to deploy" demo)
• Most ML framework focus on the “training” phase which doesn’t fit our team needs. (exploring a dataset, testing and comparing models, quick and dirty model deploy, re-deploying old models for comparaison, etc.)
• Most ML framework are focused on specific popular libraries and not on free python code. (serving any sklearn, tensorflow or h20 model)
• Most ML serving framework are "Proof of concept" oriented (cf. the usual 5min serving demo of a sklearn model on AWS) but cannot adapt to the specific constraints of most real production (multi-cloud, multi-tenant, custom infrastructure, etc.), security processes and infrastructure cost optimization.
• Most framework covers many MLOps processes instead of being minimal. Driven by their business model, most frameworks grow and become too complex to be used in a simple and transparent way.
• Most framework are “UI-intensive”, whereas we only want a python library.

What we build: modelkit

modelkit is a minimalist MLOps library for Python, built for people who want to deploy ML models to production.

It packs several features which make your go-to-production journey a breeze, and ensures that the same exact code will run in production, on your machine, or on data processing pipelines.

Wrapping your prediction code in modelkit instantly gives access to all features:

• fast: Model predictions can be batched for speed (you define the batching logic) with minimal overhead.
• composable: Models can depend on other models, and evaluate them however you need to
• extensible: Models can rely on arbitrary supporting configurations files called assets hosted on local or cloud object stores (we currently support AWS, GCP and Azure)
• type-safe: Models’ inputs and outputs can be validated by pydantic, you get type annotations for your predictions and can catch errors with static type analysis tools during development.
• async: Models support async and sync prediction functions. modelkit supports calling async code from sync code so you don't have to suffer from partially async code.
• testable: Models carry their own unit test cases, and unit testing fixtures are available for pytest or any other test framework
• fast to deploy: Models can be served in a single CLI call using fastapi

In addition, you will find that modelkit is:

• simple: Use pip to install modelkit, it is just a Python library.
• robust: Follow software development best practices: version and test all your configurations and artifacts.
• customizable: Go beyond off-the-shelf models: custom processing, heuristics, business logic, different frameworks, etc.
• framework agnostic: Bring your own framework to the table, and use whatever code or library you want. modelkit is not opinionated about how you build or train your models.
• organized: Version and share you ML library and artefacts with others, as a Python package or as a service. Let others use and evaluate your models!
• fast to code: Just write the prediction logic and that’s it. No cumbersome pre or postprocessing logic, branching options, etc… The boilerplate code is minimal and extensible.
• easy to optimize: Use our profiler to get clear sense of how to make your code faster and more efficient.

Install with pip:

pip install modelkit

Going Open Source

The move to open source was a natural step for us. We were proud of our library and thought it could be very useful to other teams.

Unlike many ML frameworks, we don’t have a business model in mind, our goal is only to see other teams benefit from our work and potentially improve it.

If you recognize yourself in the team’s context or want to know more about the project, be sure to check out the documentation and this blog post about making ML models production-ready with modelkit.

We’re looking for feedback, please contact me or join the discord https://discord.gg/ayj5wdAArV

Thanks ;-)