The Journey to Streamlining our ML Platform Interface Using our CLI Tool

Photo by Mathew Schwartz on Unsplash

In the ever-evolving world of Machine Learning Operations (MLOps), efficiency and productivity are crucial for organisations looking to capitalise on the potential of their machine learning models. The aim is to provide platform users with a streamlined and standardized interface that allows them to direct their focus towards delivering value proficiently while platform offerings are continually refined, and underlying infrastructure is optimally enhanced. To meet these needs the MLOps team at Virgin Media O2 (VMO2) developed a powerful Command-Line Interface (CLI) tool designed specifically for executing common ML tasks.

In this blog post, we will explore the advantages of the CLI tool and the journey that led to its creation. As a platform team with over 15 projects comprising approximately 50 models in production across multiple teams, the MLOps team maintains an ML platform, described in our previous blog post, which streamlines and optimises the end-to-end machine learning lifecycle, from development to deployment and beyond.

Executing command operations

In the typical workflow for executing Python scripts, you begin by navigating to the directory housing your Python script within your terminal or command prompt. Thereafter, you enter the pythoncommand followed by the designated script name, such as python main.py

In most scenarios, the process of executing commands is not as straightforward due to the necessity of command-line arguments. Consider a scenario where specific arguments like input and output files, along with a designated operation, are prerequisites for the script’s functionality. Consequently, the command evolves into a more intricate form:

python main.py --input input.txt --output output.txt --operation train

As the command’s length expands, developers are faced with the daunting task of remembering an increasing number of keywords, increasing the likelihood of errors and typographical mistakes.

Makefiles to the rescue

To eliminate mistakes as describe above, and to automate the execution of repeated tasks, we initially adopted the use of Makefiles. Makefiles are a special type of file used for automating build processes and managing dependencies of a software project. This helps to streamline tasks like compiling, testing, packaging, and other repetitive tasks.

In the Makefile, a list of rules is highlighted alongside the target name for each rule. With defined rules, targets and commands, Makefile executes specified commands using the make command followed by the target name. For instance, rather than listing a long line of Python dependencies, an install task could be created to install all available dependencies in a requirements.txt file by simply running a make install command. An example of a simple Makefile for a Python project is illustrated below.

venv: # Set up virtual environment
  python -m venv venv
install: # Install dependencies
  pip install -r requirements.txt
test: # Run tests
  pytest -v tests/
docs: # Build documentation
  sphinx-build docs/ docs/_build
package: # Package the project
  python setup.py sdist bdist_wheel

To run those commands, a Makefile is created in the root of the directory and executed with the make command accordingly.

Makefile serves as a favourable substitute for numerous bash scripts, streamlining the automation of laborious commands. Coupled with fancy descriptions indicating the explanation purposes of each command in form of comments. They can significantly simplify development and execution process by providing a standardised way to manage tasks projects, but this is not without its drawbacks.

Drawbacks of Makefiles

While the Makefile approach demonstrates efficacy in managing smaller projects, its suitability diminishes as projects expand in size and complexity. The main shortcoming lies in its scalability, particularly evident when attempting to implement an identical build system across multiple similar projects. In such scenarios, reproducing a Makefile from one project to another entails duplicating it in each new project directory.

Additional drawbacks can arise with the introduction of new package features because it necessitates the creation of new trigger functions. Given that these new features are required across all existing projects, this implies modifying over multiple distinct Makefiles. This form of manual modification does not follow software engineering best practices.

Here are the major drawbacks of using Makefiles in a large project:

1. Maintenance overhead — As the number of projects increases, maintaining multiple copies of the same Makefile becomes cumbersome. If a change is required in the build process or dependencies, you will need to apply it individually to each Makefile, thus introducing code duplication, which can be error-prone and time-consuming.
2. Inconsistent project interfaces — Copying Makefiles can lead to inconsistencies between projects, especially if modifications are made to individual Makefiles. Inconsistent build systems may cause different projects to behave differently, making it challenging to ensure uniformity and predictability across the codebase.
3. Difficulty in scaling complex projects — As a project grows more intricate, the Makefile might become convoluted and challenging to manage. Complex projects often require more advanced build systems or dependency management tools, and a basic Makefile might not be sufficient.
4. Lack of internal tooling for Bash — The platform tooling is primarily written in Python. The team has mature strategies and tooling for managing the lifecycle of Python based applications which couldn’t be reused when creating Makefiles.

To address the above-mentioned issues, the MLOps team created a Python based CLI.

The custom MLOps CLI solution

The MLOps CLI offers a unified interface for all common MLOps tasks, reducing the need for context switching and improving overall efficiency. It allows users to define, manage, and execute machine learning pipelines as a series of cohesive and reusable steps. Users can easily trigger pipelines with a single command, automating the entire process.

The CLI allows users to create custom configurations for different ML pipeline scenarios, tailoring the execution process to specific project requirements. This flexibility enhances the adaptability and reusability of the tool across various projects. It provides comprehensive functionality to manage Vertex Workbench instances seamlessly, such as creating, starting and stopping instances.

Figure 1: Executing commands via custom CLI. All project repositories are now relying on a single CLI package to run commands, bringing consistency to the user experience.

The MLOps CLI is packaged as a Python package, leveraging the Poetry dependency management and packaging tool. The package was built using Python and Click. To facilitate easy installation and distribution, the CLI is published to a private GitLab Package Registry. The GitLab CI/CD pipeline is set up to automatically build and push the package to the registry whenever changes are pushed to the repository. This ensures that the latest version of the package is readily available for users.

With the MLOps CLI, users can bid farewell to the complexities of Makefiles. It abstracts away the technical details, providing a user-friendly interface that significantly streamlines pipeline management. Figure 1 shows how the platform users now interact with multiple projects through a unified interface, reducing replication of common commands and minimising friction when added new features.

Figure 2: Subset of target trigger options in the CLI. The CLI is utilising Click’s command nesting, allowing for a functional grouping of available commands and their descriptions, providing an intuitive user experience.

Implementation Demo

As displayed in Figure 2, the CLI uses the Click package’s concept of arbitrary command nesting. Since the entry command for the CLI is the keyword mlops, a group invocation is created with the mlops keyword as follows:

@click.group()
def mlops() -> None:
  """MLOps CLI Commands"""
  pass

Then a subcommand, for example pipelines, would be created as:

@click.group()
def pipelines() -> None:
  """Pipeline commands"""
  pass

The pipelines subcommand is then added to the mlops group using:

mlops.add_command(pipelines)

This approach is followed to create the hierarchy as described in Figure 2. To execute the testtrigger via the CLI, the following command would be executed, starting from the entry keyword, mlops, to the final trigger keyword through the subcommands:

mlops pipelines training test

Advantages of the MLOps CLI

• Simplified workflow — The CLI tool offers a simplified, centralised, and streamlined workflow for common ML tasks. By providing a command-line interface tailored to the needs of machine learning practitioners, the CLI enables users to execute complex tasks with a few simple command triggers. This simplification reduces the learning curve for new users and allows them to focus on their core tasks, such as model development and experimentation.
• Unified interface across projects — End users now have a consistent experience across projects when running common ML tasks. Teams can seamlessly transition between various stages of the machine learning lifecycle, ensuring efficient collaboration and rapid iteration.
• Integrated with existing deployment ecosystem — The CLI is now integrated into the existing mature internal Python ecosystem, making it easier to safely ship new features through versioned package releases. This capability ensures release stability by allowing users to depend on a specific package version, facilitating reproducibility across different environments.

Conclusion

Having a unified interface has shown to significantly enhance efficiency, productivity, scalability, and collaboration among machine learning practitioners. VMO2’s MLOps CLI tool empowers teams by providing a simple and consistent interface for common ML tasks. The addition of the CLI has streamlined machine learning workflows for our users, reduced the operational overhead of the platform, and shifted our users’ focus onto what matters most — delivering innovative and impactful AI solutions with customised integration.

Written by Ariyo Oluwasanmi with contributions from the wider VMO2 MLOps team: Alex Jones, Berk Mollamustafaoglu, Gleb Lukicov, Shehed Girsi, and Sidney Radcliffe.

Some of the MLOps team here at Virgin Media O2

We’re hiring! Want to work for us? Check out our open roles here.