AI Lifecycle Management for Virtual Assistants
As developers of Watson Assistant we have developed best practices for creating and maintaining high-performance assistants for ourselves. This guide breaks down these practices into an easy-to-understand lifecycle that you can follow for your own purposes.
As engineers who made Watson Assistant, we know it has the capabilities you need to build highly effective AI solutions for your business. But features alone are not enough. You also need to implement a process to create, analyze, and continuously improve assistants.
This guide will explains our recommendations for an AI lifecycle and the tasks involved in each phase. It’s based on our expertise gained from real world engagements with clients around the world. And now we’ll teach you what we know.
Phases of the AI lifecycle
The AI lifecycle has six phases. The phases are repeated to form an iterative process that you follow to build, manage, and incrementally improve your assistant
The six phases are illustrated in the figure below
Phase 1: Create and update training data
In this phase of the lifecycle, you create a new assistant or update an existing one. This guide shows you the steps to create one. After you launch your assistant, you need to continuously improve it as your business needs change. By following the process in the development process guide, you can turn the questions your end-customers frequently ask into a business use case, in which an assistant works in sync with a human agent. Using the “Try It Out” panel, you can then test your assistant. Submit utterances that represent reasons for contacting support and see whether you assistant returns the correct intent.
In this phase, it’s normal for some utterances to be mapped to the correct intents while others won’t. The next phase of the lifecycle introduces additional tools that can help you improve the initial design of your virtual assistant.
Phase 2: Analyze, train, and test data
In this phase, you analyze your assistant by using the Dialog Skill Analysis Notebook. You use insights generated from the data to make updates. You can use the notebook to analyze characteristics of your data, such as the number of training examples for each intent or the terms associated with a specific intent. We have found that reviewing this analysis can help you uncover potential pitfalls in the design of your dialog skill.
To perform quantitative analysis you need to create a test set. A test set is just additional data that contain examples of utterances for different intents. This is also known as a blind test set. As the name implies this data should not be the same as the examples provided during creation of the dialog skill. These could be additional examples drawn from logs, or created by subject matter experts. Ideally, for every 10 examples provided as training data for an intent, you will provide an additional 2 or 3 examples for testing that intent.
Use the test set with the notebook to generate quantitative metrics about how the dialog skill’s machine learning models perform. Remember that it is important that examples provided in the test set are what you’d see in the actual use case.
Phase 3: Deploy
To reach this point, you have created an assistant, analyzed its performance, and have updated it several times based on testing. Now you’re ready to begin the deployment processes.
In this phase, we recommend that you read the deployment documentation carefully. Reviewing the section on how to call out to your virtual assistants from within your use cases using the Watson Assistant APIs can help with the integration of the dialog skill into the final business process. Documentation related to Software Development Kits (SDK) currently supported in various programming languages can also be found there.
Phase 4: Measure live system
As your assistant runs in production and users interact with it, you can leverage the data produced to find opportunities for further improvement. The Continuous Improvement Best Practices Guide introduces the concept of measuring both the effectiveness and the coverage of your virtual assistant, and how you can use these metrics to prioritize your effort.
The Measure Notebook provides a set of automated metrics that help you monitor and understand the behavior of your live system using your conversation logs as input. The goal is to understand where your assistant is doing well and where it isn’t, and to focus your improvement efforts. For example, you might identify user utterances that aren’t covered, or which intents lead most often to abandoned conversations.
As an output of the measure notebook, you can also generate a subset of problematic conversations to be further assessed and analyzed by your team in the next phase.
Phase 5: Analyze effectiveness and coverage
With measurements of your live system in hand, you can focus on understanding the patterns that lead to low performance. If you decide to improve the coverage measurement, you’ll want to use the Intent Recommendations feature in Watson Assistant, which taps your logs to help create new intents or expand existing ones.
If you turn your attention to the effectiveness measurement, you can choose to analyze the quality of intent and entity detection using the Analyze Effectiveness Notebook or focus instead on the quality of your dialog flows using the Dialog Flow Analysis Notebook.
This analysis phase helps you prioritize the next steps of your improvement effort.
Phase 6: Improve via recommendations
You make further improvements to your virtual assistant with analysis provided by the Effectiveness Notebook or the Dialog Flow Analysis Notebook, or the insights provided by the Dialog Skill Analysis Notebook. These improvements can include resolving intent conflicts, adding training to imprecise intents, or combining confused intents into a single intent and distinguishing using entities. As you update your training, you’ll start Phase 1 of the next iteration of your AI lifecycle.
We hope that the guidance provided here helps you on your conversational AI journey with Watson Assistant. We also welcome feedback based on your experience designing, deploying, and maintaining an assistant that supports your business process.
