A Version Control Strategy for AI Applications

Libraries are a great example of version control

This document describes a source control approach for AI projects. Our exemplary project is a voice application which uses three Watson services: Watson Assistant, Speech to Text, and Text to Speech. Our exemplary infrastructure has four regions: Development (DEV), System Integration Test (SIT), Quality Assurance (QA), and Production (PROD). This document describes a to implement concurrent streams of development and how these streams will be deployed through each of the four environments.

Concurrent development

The concurrent development branching strategy described here is based on GitFlow and Branch per Environment (BpE). This strategy includes four long-lived branches, one for each of the major environments in our infrastructure.

For day-to-day activities developers branch off of and merge into the dev branch. When code needs to be deployed into higher environments the following actions happen.

· The code is tagged.

· The code is merged from the lower branch to the higher branch

· Code from the higher branch is deployed to the new environment and tested

A general depiction of these activities is included in the image below. This image shows the progression of various major releases (Mn) and new features (Fn) through various branches and environments.

General branching strategy for AI application

Deployments with Watson Assistant

Deploying updates to higher Watson Assistant environments is performed by updating a series of Watson Assistant workspaces. Each workspace is exportable to and importable from a single JSON file. A deployment is simply to take a series of source workspaces from a lower environment and overwrites a series of target workspaces on the higher environment

The following steps should generally be performed to deploy Watson Assistant code to a higher environment:

· Download each workspace file with get workspace API. Use flag export=true. For ease of comparison across versions additionally use flag sort=stable.

· Back up current workspace(s) in target environment.

· Compare backed-up workspace(s) to what is stored in the branch associated with this environment. Ensure there are no changes.

· Push the changes from lower environment and overwrite in higher environment with the workspace JSON update workspace API.

· Perform one or more smoke tests. In case of any issues, roll back to the backed up workspaces using the update workspace API.

· When deployment is successful, tag code and merge up to higher branch.

Deployments with Speech to Text

A Speech to Text model is trained from a set of files that should be in version control.

Language Models are trained from a collection of text files, also called corpora. Acoustic Models are trained from a collection of audio files and text files, the text files being the text spoken across all of the audio files.

Overview of Speech to Text training assets

The following steps should generally be performed to deploy a Speech to Text model to a higher environment:

· Create a branch representing the training data used to train the model you wish to promote

· Upload new training data

· Call the training API for the model

· Perform one or more smoke tests. In case of any issues, roll back to the previous model

· When deployment is successful, tag code and merge up to higher branch

Language model updates more specifically include the following steps:

· Add new training corpus files with language model add corpus API

· Train language model with language model training API

· If rollback is needed, remove new corpus files with language model delete corpus API — or delete all corpus files and use add corpus API on the pre-update corpus files

· Call the language model training API

Acoustic models are trained with audio files and a targeted language model. The files in this targeted language model include only the text spoken in the audio files and no other textual content. Thus acoustic model updates often include two parts, an update of the targeted language model and an update of the acoustic model itself. The targeted language model is updated (and rolled back) as described above.

Acoustic model upgrades more specifically include the following steps:

· Add new corpus files to the targeted language model and train the language model

· Add new training audio files with acoustic model add audio API

· Train acoustic model with acoustic model training API

· If rollback is needed, remove new audio files with acoustic model delete audio API — or delete all audio files and use add audio API on the old audio files

· Call the acoustic model training API

Deployments with Text to Speech

Text to Speech deployments are used to update dictionaries used by Text to Speech. These dictionaries contain pronunciation guides to out-of-vocabulary words, including domain jargon and acronyms.

The following steps should generally be performed to deploy a Text to Speech model to a higher environment:

· Create a branch representing the dictionary you wish to promote

· Run the dictionary update API command with the dictionary specified

· Perform one or more smoke tests. In case of any issues, roll back to the previous dictionary by running the update API command with previous dictionary

· When deployment is successful, tag code and merge up to higher branch

Hotfix strategy

If a bug is identified in production and the fix must be deployed before the next scheduled major release a hotfix strategy can be used. In this case a new branch is created immediately from the ‘prod’ branch and this code is immediately deployed on top of the ‘dev’ environment.

When the bug is fixed, two major activities happen:

1. The code on this branch is deployed to higher environments and tested as usual

2. The code on this branch is merged down into the ‘dev’ branch.

This revised workflow is demonstrated below:

Hotfix branching strategy for AI applications

Conclusion

This approach shows you how to use version control with several different Watson APIs for both daily development as well as promotion from development to production environments. The benefits of version control are well known and apply well to Watson solution development assets. The overall approach described in this document will be applicable to several other Watson services as well after the specific commands are adapted. Even if you don’t follow these exact steps I strongly urge you to use version control in your AI projects — it is just as important to track your AI assets as you do any other code asset.