Creating a MLOPS pipeline with Azure Machine Learning Service

In the previous articles we looked at the code needed to prepare data and train a machine learning model in Azure ML. In this article I will show how to take this code base and create a pipeline job that can run on an Azure ML cluster.

Model life cycle

Now that we have code to prepare the data, train and score the model as well as save the model to a location in Azure we can go to the next step: creating a pipeline.

In this pipeline we will be running the steps I just mentioned in the same order as seen here:

Model training pipeline in Azure ML

Before we get started let’s first look at the yaml file for a component in Azure ML:

jobs:
  component_prep_data:
    type: command
    component: ./components/prep.yml
    inputs:
      cfg: ${{parent.inputs.cfg}}
      raw_data: ${{parent.inputs.raw_data}}
      enable_monitoring: ${{parent.inputs.enable_monitoring}}
      table_name: ${{parent.inputs.table_name}}
    outputs:
      data_path: ${{parent.outputs.data_path}}

The component calls the following job prep.yml with the necessary inputs and outputs:

$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
type: command

name: component_prep_data
display_name: prepDataComponent
version: 1

inputs:
  raw_data:
    type: uri_file
  enable_monitoring:
    type: string
  table_name:
    type: string
  cfg:
    type: uri_file

outputs:
  data_path:
    type: uri_folder

code: ../../src/mlops/

environment: env_name:env_version

command: >-
  python prep.py
  --raw_data ${{inputs.raw_data}}
  --enable_monitoring ${{inputs.enable_monitoring}}
  --table_name ${{inputs.table_name}}
  --cfg ${{inputs.cfg}}
  --data_path ${{outputs.data_path}}

The job prep_data:

• Inputs: files and folders that we will be reading from (read-only)
• Outputs: files and folders we will be writing to (over-write is not an option -> make the paths unique with every run)
• The code will be deployed from my local system and the path to the code base is relative to the pipelines folder: ../src/mlops
• I use a custom environment based on openmpi4.1.0-ubuntu20.04
• We run the python file prep.py with the necessary parameters

The entire pipeline:

$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
experiment_name: model-training
description: Training Pipeline to train a model

inputs:
  raw_data:
    mode: ro_mount
    type: uri_file
    path: azureml://datastores/xxxxxx/paths/credit_defaults_model/data/raw/default_credit_card_clients.csv
  enable_monitoring: 'false'
  table_name: 'creadit_card_defaults_table'
  cfg:
    mode: ro_mount
    type: uri_file
    path: azureml://datastores/xxxxxx/paths/credit_defaults_model/config/config.yml

outputs:
  data_path:
    mode: rw_mount
    type: uri_folder
    path: azureml://datastores/xxxxxx/paths/credit_defaults_model/data
  evaluation_output:
    mode: rw_mount
    type: uri_folder
    path: azureml://datastores/xxxxxx/paths/credit_defaults_model/1/score
  model_info_output_path:
    mode: rw_mount
    type: uri_folder
    path: azureml://datastores/xxxxxx/paths/credit_defaults_model/1/model
  trained_model:
    type: custom_model
    path: azureml://datastores/xxxxxx/paths/credit_defaults_model/1/model

settings:
  default_datastore: azureml:xxxxxx
  default_compute: azureml:training-cpu
  continue_on_step_failure: false
  default_environment: azureml:env_name:env_version

jobs:
  component_prep_data:
    type: command
    component: ./components/prep.yml
    inputs:
      cfg: ${{parent.inputs.cfg}}
      raw_data: ${{parent.inputs.raw_data}}
      enable_monitoring: ${{parent.inputs.enable_monitoring}}
      table_name: ${{parent.inputs.table_name}}
    outputs:
      data_path: ${{parent.outputs.data_path}}

  component_train_model:
    type: command
    component: ./components/train.yml
    inputs:
      data_path: ${{parent.jobs.component_prep_data.outputs.data_path}}
      learning_rate: 0.75
    outputs:
      model_output: ${{parent.outputs.trained_model}}
  
  component_evaluate_model:
    type: command
    component: ./components/evaluate.yml
    inputs:
      cfg: ${{parent.inputs.cfg}}
      data_path: ${{parent.jobs.component_prep_data.outputs.data_path}}
      model_path: ${{parent.jobs.component_train_model.outputs.model_output}}
    outputs:
      output_folder: ${{parent.outputs.evaluation_output}}
      
  component_register_model:
    type: command
    component: ./components/register.yml
    inputs:
      cfg: ${{parent.inputs.cfg}}
      model_path: ${{parent.jobs.component_train_model.outputs.model_output}}
      evaluation_output: ${{parent.jobs.component_evaluate_model.outputs.output_folder}}
    outputs:
      model_info_output_path: ${{parent.outputs.model_info_output_path}}

Once we have run this pipeline in Azure ML we can re-run or schedule it to run using the Azure ML UI. We can also re-run the job with different parameters (input and output) directly in the UI so it’s important to choose those wisely.

To use the parameters in the different jobs you reference them as parent.inputs.parameter or parent.outputs.parameter. To reuse outputs from previous jobs you use parents.jobs.job_step.outputs.parameter.

Components:

Train-model component:

$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
type: command

name: component_train_model
display_name: trainModelComponent
version: 1

inputs:
  data_path:
    type: uri_folder
  learning_rate:
    type: number

outputs:
  model_output:
    type: uri_folder

code: ../../src/mlops/

environment: azureml:env_name:env_version

command: >-
  python train.py
  --data_path ${{inputs.data_path}}
  --learning_rate ${{inputs.learning_rate}}
  --model_output ${{outputs.model_output}}

Evaluate-model component:

$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
type: command

name: component_evaluate_model
display_name: evaluateModelComponent
version: 1

inputs:
  data_path:
    type: uri_folder
  cfg:
    type: uri_file
  model_path:
    type: uri_folder

outputs:
  output_folder:
    type: uri_folder

code: ../../src/mlops/

environment: azureml:env_name:env_version

command: >-
  python evaluate.py
  --data_path ${{inputs.data_path}}
  --cfg ${{inputs.cfg}}
  --model_path ${{inputs.model_path}}
  --output_folder ${{outputs.output_folder}}

Register-model component:

$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
type: command

name: component_register_model
display_name: registerModelComponent
version: 1

inputs:
  cfg:
    type: uri_file
  evaluation_output:
    type: uri_folder
  model_path:
    type: custom_model

outputs:
  model_info_output_path:
    type: uri_folder

code: ../../src/mlops/

environment: azureml:env_name:env_version

command: >-
  python register.py
  --model_path ${{inputs.model_path}}
  --evaluation_output ${{inputs.evaluation_output}}
  --model_info_output_path ${{outputs.model_info_output_path}}
  --cfg ${{inputs.cfg}}

Creating a custom environment

Before we can run the above pipeline we need to create a custom environment. For this we run another job using the following conda.yml:

channels:
  - defaults
  - anaconda
  - conda-forge
dependencies:
  - python=3.8.6
  - pip<=21.3.1
  - pip:
      - matplotlib==3.5.0
      - psutil==5.8.0
      - tqdm==4.62.0
      - pandas==1.3.0
      - scipy==1.7.0
      - numpy==1.21.0
      - ipykernel==6.0
      - azureml-core==1.51.0
      - azureml-defaults==1.51.0
      - azureml-mlflow==1.51.0
      - azureml-telemetry==1.51.0
      - azure-ai-ml==1.9.0
      - scikit-learn==1.0.0
      - debugpy==1.6.3
      - fsspec==2021.8.1

The create-environment.yml looks like this:

$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: environment_name 
image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:20230608.v1
conda_file: ../../environment/conda.yml
description: Environment created from a Docker image plus Conda environment.

Summary

In this article we put the previous code together to create a complete training pipeline in Azure ML. By using components we created small code blocks that can be reused. We created a custom environment to run our code in. Finally we have seen that by using the Azure ML tools we can create a machine learning pipeline that could be reused for more complex data science projects.

In the next article I will be showing you how to create an endpoint in Azure ML, deploy your model and call the endpoint from a simple web app. We will be coding step 4 of the model lifecycle.

If you found this article useful, please follow me.