Managing ML Training Models using ModelDB

Version, Track, Collaborate On, and Query Your Machine Learning Training Experiment Runs

Published in

Hashmap, an NTT DATA Company

5 min readJun 28, 2019

Building and improving a machine learning (ML) training model’s performance is an iterative process and often involves trial and error. Some of the most common adjustments are changing the classifier, adding preprocessing steps, and adjusting hyper parameters.

For example, if 20–30 ML training model runs were done from a certain starting point, it can be challenging not to lose track of that starting point, what changes took place during the various runs, and what the impact of those changes was for each step. In short, it’s highly likely that key insights will be lost from the experiment runs performed without some level of management.

Thankfully, there is a tool that helps in solving these challenges — it’s ModelDB.

In this post, I’ll take a look at ModelDB along with the benefits and use cases it caters to as well as step through a sample implementation. To get the most value out of what I am sharing, I would recommend having a basic knowledge of ML concepts and familiarity with toolkits like scikit-learn, jupyter notebook, docker, etc. to understand the setup and sample code walkthrough below.

ModelDB — What is it?

ModelDB makes it easier to manage ML training models, and provides a capability to store, share, query, visualize and version those models which allows the tracking of model development. It is specifically helpful when debugging and tuning models, as it stores model metadata and metrics of all the experimental runs of the model which can be easily referred to and compared using visualization features provided by ModelDB.

It is best used for use cases such as:

Tracking Modeling Experiments
Versioning Models
Ensuring Reproducibility
Visual exploration of models and results
Collaboration

ModelDB Architecture

Source: https://mitdbg.github.io/modeldb

Setting Up ModelDB

The easiest way to get ModelDB going locally is to use docker-compose for Docker based setup. Just clone the GitHub repository ModelDB and inside the root folder of the repository run

docker-compose up

This will build docker images for all the required services and bring them up. Please refer to the docker-compose.yml file in the repository for more details.

Other setup options are described in detail in the documentation on GitHub.

What Does ModelDB Natively Support?

As of now, ModelDB has a native support for scikit-learn and spark.ml. The native support makes it really easy to track training metadata of the models without any significant change in the model code. It automatically captures and stores the metadata of all the training stages like transform, fit and predict in the background.

However, we lose the flexibility to structure and log metadata in a unique way for a specific use case. Adding to that, there is no native support for deep learning frameworks like TensorFlow Keras, but there is an option of using ModelDB’s light API to overcome these limitations.

ModelDB Light API

The ModelDB python client library provides APIs which can be used with any ML workflow to log model metadata and metrics. First things first — prepare the metadata, create syncer object and sync the data to ModelDB. Let’s see this in detail with the help of an example.

Face Recognition Example Walkthru

For this example demonstrates training a model to predict faces from The Labeled Faces in the Wild face recognition dataset using scikit-learn. More details are here.

Download the lfw data of sklearn dataset, if not already on disk and load it as numpy arrays.

from sklearn.datasets import fetch_lfw_peoplelfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)
X = lfw_people.data
y = lfw_people.target

Split data into training and test sets.

from sklearn.cross_validation import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

Compute a PCA on the face dataset for dimensionality reduction.

from time import time
from sklearn.decomposition import RandomizedPCApca = RandomizedPCA(n_components=150, whiten=True).fit(X_train)
X_train_pca = pca.transform(X_train)
X_test_pca = pca.transform(X_test)

Train an SVM classification model.

from sklearn.svm import SVC
from sklearn.grid_search import GridSearchCVparam_grid = {‘C’: [1e3, 5e3, 1e4, 5e4, 1e5], ‘gamma’: [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1], }svc_kernel = ‘linear’clf = GridSearchCV(SVC(kernel=svc_kernel, class_weight=’balanced’), param_grid)clf.fit(X_train_pca, y_train)

Do a quantitative evaluation of the model quality on the test set.

y_pred = clf.predict(X_test_pca)from sklearn.metrics import accuracy_score
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error
from sklearn.metrics import precision_scoreaccuracy_score = accuracy_score(y_test, y_pred)
mae_score = mean_absolute_error(y_test, y_pred)
mse_score = mean_squared_error(y_test, y_pred)
precision_score = precision_score(y_test, y_pred)

The model is trained and we have computed some evaluation metrics for it. Let’s sync these to ModelDB using Light API.

Initialize the ModelDB Syncer object.

from modeldb.basic.ModelDbSyncerBase import *syncer_obj = Syncer(
    NewOrExistingProject("ModelDB Light API POC", "anuj", "Learn to       
      use Model DB Light API"), 
    NewOrExistingExperiment("Face Recognition", "Face Recognition"),      
    NewExperimentRun("Trial Run")
)

Initialize dataset object with data and metadata and store in ModelDB — Create the Model, ModelConfig, and ModelMetrics instances.

data_folder_path = “/Users/anuj/scikit_learn_data/lfw_home/lfw_funneled”datasets = {
 “train” : Dataset(data_folder_path, {“num_cols” : X_train.shape[1],    
                   “dist” : “random”}),
 “test” : Dataset(data_folder_path, {“num_cols” : X_test.shape[1],
                   “dist” : “random”})
}mdb_model = Model(“Classification”, “face_recognition”)model_config = ModelConfig(“Classification”, {“svc_kernel” :
                        svc_kernel, “param_grid”:param_grid})model_metrics = ModelMetrics({“accuracy”:accuracy_score,
    “mean_squared_error”:mse_score, “mean_absolute_error”:mae_score,
    “precision_score”:precision_score}
)

Sync the data to ModelDB.

syncer_obj.sync_datasets(datasets)
syncer_obj.sync_model(“train”, model_config, mdb_model)
syncer_obj.sync_metrics(“test”, mdb_model, model_metrics)syncer_obj.sync()

That’s it! You have now successfully trained a model for recognizing faces from the lfw dataset and tracked the model’s metadata and metrics in ModelDB. The complete code sample is available here.

Tune and Retrain the Model

Adjust the model parameters and retrain the model to achieve a better accuracy score — each experimental run will sync the corresponding data to ModelDB. The ModelDB UI allows exploration and comparison of the metrics and metadata for all the runs.

Screenshots are below…

Conclusion

I hope you’ve enjoyed this walkthru of ModelDB — for exploring it in more detail, go ahead and play around with the ModelDB visualizations.

In general, I’ve found that ModelDB has a very useful feature set and it’s relatively easy to configure and use, but it is still evolving, and will continue to improve. In the near future, I hope to see native support for additional ML toolkits added to ModelDB.