Continuously Improve Your Watson Assistant with Jupyter Notebooks
So, you built a virtual assistant, released it to users, and you’re excited to watch as its conversation logs grow. Now you have all sorts of real user conversations that you can use to improve your next version of the assistant. However, you face a common problem: How can I use this large number of conversations to improve my virtual assistant efficiently?
[TLDR: IBM is providing two Jupyter notebooks to help you continuously improve your virtual assistant. Learn more here.]
When you first release an assistant, low usage volumes make it possible to inspect every user conversation to find problems and identify areas for improvement. But the manual inspection approach does not scale as conversation volumes grow. The Watson Assistant team built two Jupyter notebooks that can help you maximize the training power of your conversations in less time and with fewer resources. These Jupyter notebooks contain pre-built Python code that someone with minimal coding experience can run to generate helpful analytics from your data. The results are displayed in visualizations that make the analysis easier to comprehend.
The Watson Assistant Measure Notebook focuses on the overall performance of the virtual assistant. The Watson Assistant Effectiveness Notebook identifies problem areas and patterns that highlight areas of your assistant that need to be improved.
Watson Assistant Measure Notebook
The Watson Assistant Measure Notebook gives you an understanding of how well your assistant is performing. It helps you decide whether to focus on improving your assistant’s coverage or its effectiveness. Coverage measures the portion of total user queries that your assistant attempts to answer. Effectiveness measures how well your assistant handles the queries it answers.
Analysis of effectiveness and coverage should complement each other. If your assistant is answering questions incorrectly, you can improve it by focusing on improving effectiveness. If your assistant is not answering enough questions, you can improve it by focusing on expanding coverage. The Watson Assistant Measure Notebook provides the flexibility to customize how you define coverage and effectiveness for your assistant.
Watson Assistant Effectiveness Notebook
The Watson Assistant Effectiveness Notebook helps you understand exactly what areas to focus on to improve your assistant’s effectiveness. It summarizes the intents that perform the worst, the pairs of intents that your assistant has trouble distinguishing between, and the dialog nodes and entities that commonly cause ineffective conversations. These summaries show you where to focus your improvement efforts, and what you need to do to get better results.
Get Started
These notebooks have the potential to drastically reduce the time and maximize the impact of your improvement effort. We encourage you to use these notebooks throughout the improvement lifecycle. As you have more and more conversations to sift through, the notebooks make it easier to optimize what you can learn from them.
To get started, you can run both notebooks for free in Watson Studio, or locally by using stand-alone Python code. To run them in Watson Studio, follow these instructions. To run them locally, get the notebooks from this IBM GitHub Repository. You can learn more about how to improve your assistant by reading the continuous improvement best practices.
Resources
Technical documentation on Measure and Effectiveness Notebooks
Stand alone Measure and Effectiveness Notebooks
Watson Assistant Measure Notebook in Watson Studio
