A comparison of DevOps choices for IBM Watson Assistant

Leigh Williamson
11 min readApr 27, 2020

The goal of this article is to describe and compare various options for managing the development process for Watson Assistant projects. In particular, several strategies for using the internal Versions feature of Watson Assistant are compared with the use of external version control and release management tools.

The article begins by examining some fully self-contained strategies which only require features found within the Watson Assistant service itself and are very well described in Mitch Mason’s earlier article. Then more complex strategies are evaluated which leverage tools and procedures outside of the Watson Assistant service.

The strategies described in this article are not all mutually exclusive. Some can be used in combination and such combinations may offer the best balance between quick live updates and more formal (and more time consuming) stable releases.

A list of pros and cons for each strategy are provided at the end of each section. The article is written with an assumption that the reader is already familiar with Watson Assistant service concepts and also familiar with various general purpose DevOps tools available in the market. Daniel Toczala has published a great article about this also and I highly recommend that you read that article in conjunction with this one.

Self-contained Watson Assistant strategies

One Skill, multiple versions

The easiest and most simple strategy is to use a single Dialog Skill for both development and production (and testing also).

Within the Watson Assistant service web based tooling, you can create “snapshot” versions of the Dialog Skill which controls the flow of conversation for the virtual assistant. It’s a good practice to create a checkpoint version of the Dialog Skill as you develop it and reach important functional milestones. After you create a new version “snapshot”, it remains “frozen” as-is and no further changes can be made to it. All future changes to the Skill occur in the Development version, at the head of the version list. When you get ready to release an update for your virtual assistant, it is a best practice to create a “release version” and not to use the Development version for live production purposes.

The Watson Assistant web widget code, that is embedded in web pages where the virtual assistant is exposed, interacts with the Assistant “object” in the IBM Cloud Watson Assistant service instance. And the Assistant object can be linked to any of the internal versions defined for the Dialog Skill. Furthermore, the Assistant’s linkage can be dynamically changed to use a different version of the same Dialog Skill. Such changes to the version of the Dialog Skill used by the Assistant are instantaneous and reflected in user interaction with the web widget immediately.

The most simple strategy for Dialog Skill version control involves declaring that a known good version of the Dialog Skill is to be the new “Production” version and linking the Assistant (and by extension the web widget) to that version of the Skill. Typically, this version is labeled so that it’s easy to identify as the production one in the list of Skill versions.

While the web widget and Assistant object continue to point to the new Production version of the Dialog Skill, ongoing development can occur using the default (Development) version of the Skill without having any impact on the production environment. Snapshot versions of the Skill can be made from the Development environment, and these will not be reflected in the web chat. And when the Development version of the Skill contains sufficient number of enhancements and bug fixes (and is sufficiently tested), a new snap-shot version of the Dialog Skill can be made and labeled as the “new” Production version.

When it is time to make the new Production version of the Skill live, all that is needed is to change the Assistant object configuration so that it is linked to the new production version. The old production version of the Skill will remain available and if needed because of a problem in the new version, it can be used as a rapid fallback.

Pros:

  • No external tools or services required
  • Fast switch from one version of the Dialog Skill to another
  • Quick rollback available as a by-product of this approach

Cons:

  • No external backup saved — everything is within one Skill, and all inside the Watson Assistant service
  • No ability to compare one version of the Skill with another
  • No physical separation of development environment from production (it’s all contained within one Dialog Skill)

One development Skill, one production Skill

A slightly more involved strategy involves making two copies of the Dialog Skill and declaring one Skill to be the “Production” Skill and the other to be the “Development” Skill.

All new development is done using only the Development Skill and when it is decided to put new features into production, those modifications (new Intents, Entities, Dialog nodes, etc.) are copied from the Development Skill over to the Production Skill. Within the Production Skill, a new version is created when new content is copied from the Development Skill. And the Assistant object is switched to make use of the newest version in the Production Skill when it is time for the new features to go live.

The main concept for this strategy is to make use of a completely separate Dialog Skill for all development work and to reserve the other Dialog Skill to be for Production use only. A variant of this strategy is to employ multiple separate “Development” Skills which are used by different developers, much like git repositories support multiple branches off of the one master branch.

Pros:

  • Better separation of development activity from production, no chance of mixing by mistake
  • Potential for multiple developers to work on their own Skill copies without collision
  • Safer and easier to support “hot fixes” to Production Skill (but be sure to put them into the Development Skills also!)

Cons:

  • More steps involved in getting changes into production (copying fragments of a Skill is tricky)
  • Potential confusion if there are too many separate Development Skills to merge
  • No external backup or disaster recovery

Interleaved production Skills

A strategy for employing interleaved Dialog Skills also involves separate copies of the Dialog Skill, but uses them differently. In this approach the Skill to be used for the production target changes with each release and the “old” Skill becomes the new development Skill.

All development changes are put in the non-production Skill until it’s ready to be released. At that time, a version of the non-production Skill is created and labeled to be used next for production. Then the Assistant object is changed to “point” at this Skill and version, making it the new production release.

A copy of the new production Skill is made and used as the base for further development. The Skill previously used for production is “frozen” and saved in case a rapid rollback is needed. Then when the next set of features has been completed, the same “pointer swap” from the Assistant is done again.

Pros:

  • Easier (than strategy #2) to swap in new development changes to production (no copying of Skill fragments)
  • Separation of development environment from production is maintained
  • Unaltered previous production environment is saved in case rollback is needed

Cons:

  • Requires careful attention and labeling to keep track of Skills and versions
  • No external backup or disaster recovery

Leveraging External Tools

Export the Skill

Any Dialog Skill in Watson Assistant can be exported out of the service as a JSON formatted text file. And these exported Skill files can be imported into a different instance of Watson Assistant. They can also serve as an external backup of the Skill, for disaster recovery purposes. Furthermore, because they are text files, they can be stored in any of the traditional source code version control systems available such as git.

Skills can be exported and imported from a Watson Assistant service instance using either the web based graphical UI or the command line. Some of the options available from the command line are not available in the GUI, and vice versa. The command line can be automated by a deployment tool such as Jenkins and it can also be used by a periodic job scheduler such as cron. I like to use the Cloud Functions serverless technology in the IBM Cloud platform to create a periodic automated action to export Watson Assistant Skills.

To address the need for external backup and disaster recovery, many Watson Assistant project combine this ability to export the Skill contents with one of the internal Skill version strategies described in the previous part of this article. The strategy involves two steps: first the Dialog Skill is exported to a file, and then the Assistant object is switched to be linked with the new production version in the Skill.

Pros:

  • External backup / disaster recovery provided
  • Exported Skill JSON can be imported into a different Watson Assistant instance
  • Fast switch from one version of the Dialog Skill to another
  • Quick rollback available as a by-product of this approach

Cons:

  • No ability to compare one version of the Skill with another
  • No separation of development environment from production (it’s all one Skill)

Include a Git repository along with textual diff capability

Why not add an external version control system to this picture? This strategy has the added benefit of being able to display the differences between versions of the Dialog Skill. However, in order to support the “diff” capability, the Skill export must be done using the command line. See the Appendix for details about using the command line to export Dialog Skills.

Without using the special “sort=stable” command attribute when exporting a Dialog Skill, the order in which the Skill content is exported is random and the results of one export can’t be reasonably compared against the results of another export of the same exact Skill. The “sort=stable” attribute for exporting the Skill is only available when using the command line. There is no current way to export a Skill from the GUI such that the resulting JSON file can be used to compare against another exported Skill. So, you must use the command line to export Skills for the purposes of comparing them.

Example comparison between two exported Watson Assistant Dialog Skill versions

Pros:

  • External backup / disaster recovery provided, in addition to robust version control
  • Different versions of the Dialog Skill can be visually compared with each other in order to see what changed between the versions
  • More robust comments, labeling and tagging can be applied to the Skill versions

Cons:

  • No separation of development environment from production (it’s all one Skill)

Various combinations of internal versioning and external tools

Use of an external version control system can be combined with any of the self-contained strategies (for instance, the interleaved Skill strategy described earlier) to in order to obtain an enhanced combination of ease of internal version switching with the robust external backup and version comparison.

Pros:

  • External backup / disaster recovery provided, in addition to robust version control
  • Different versions of the Dialog Skill can be visually compared with each other in order to see what changed between the versions
  • More robust comments, labeling and tagging can be applied to the Skill versions
  • Separation of development environment from production Skills

Cons:

  • More steps and complexity

Automated CI/CD pipeline

At the time that this article was written, it is not possible to use other than the Watson Assistant GUI to create new Dialog Skill versions or switch the Assistant between different Skill versions. So, all of the “self-contained” techniques in this paper can only be done manually using the web UI.

However, exporting and importing of Dialog Skills, as described in the “external strategies” in this paper are possible to automate. And, in fact, you cannot perform the stable format export of a Dialog Skill other than by using the command line (which can be automated by any popular continuous delivery tool such as Jenkins or IBM Cloud Continuous Delivery service).

Many projects evolve to include the use of an automated deployment, or CI/CD, product as part of the DevOps process. And in these scenarios it is sometimes the case that the Watson Assistant service instance in which development occurs is a different instance from the one where the production Skill is running. This is referred to as a staged pipeline of service instances, progressing from initial development environment through a testing environment and finally ending in a “locked down” production environment (e.g. multiple Watson Assistant service instances).

The main reasons for adding this amount of complexity is to provide an isolation and protection of the Watson Assistant environment used for production. Access to the production instance can be restricted so that only the automated CI/CD tool can make changes and only via its tested and approved automation execution. Also, CI/CD systems produce a log of the actions executed and the results, so there is an audit trail in case anything goes wrong during the release.

And it is frequently the case that the full virtual assistant solution includes components besides just the Watson Assistant service. There may be code outside of Watson Assistant that makes API calls to the service and there may be data sources or external applications that the Watson Assistant invokes via it’s WebHook integration. For solutions that involve more than just the Watson Assistant service, use of an automated CI/CD system allows for the coordination of the release of all components in the solution together at one time.

Pros:

  • Strong isolation and protection of the Production environment
  • External backup and disaster recovery for the Dialog Skill
  • Can integrate the release of Watson Assistant changes with the release of other related code (code that makes API calls to the virtual assistant)

Cons:

  • Much more complex system to set up and maintain

Watson Assistant command line Skill export

Documentation for using the curl command to export a Skill (workspace) is available at the following link:

https://cloud.ibm.com/apidocs/assistant#get-information-about-a-workspace

You can copy an example curl command directly from the documentation page to your laptop clipboard, and paste it into the command line for execution.

The following is an example of the command to export a Skill (previously called a “workspace”) with the “sort=stable” attribute set so that the resulting file can be compared with a different exported file from the same original Skill root:

curl -u “apikey:{paste-your-apikey-here}” “https://gateway.watsonplatform.net/assistant/api/v1/workspaces/{paste-your-workspace_id-here}?version=2019-02-28&export=true&sort=stable" >export.1.json

The command above produces a file named “export.1.json” containing the sorted elements from the targeted Skill.

Conclusion

There are lots of alternatives when it comes to how you manage production releases of virtual assistant projects based on IBM Watson Assistant. You can go with as light-weight and simple an approach as possible. Or you can integrate the release management of the virtual assistant with the most sophisticated enterprise DevOps automation systems on the market today. The choice is yours and should be made based on the needs and budget for your particular project.

--

--

Leigh Williamson

Leigh is a software engineer who has been hanging around Austin Texas for a long time, while working at IBM on several of the company’s major software products.