Watson Assistant Search Skill: Taking Virtual Assistant To The Next Level

As the world reacts to the Covid-19 coronavirus pandemic, call centers of hospitals, government agencies, and other organizations are overwhelmed with the number of calls to find out repeated common information. Everyone wants personal interaction to get information from authentic sources. Remote working and collaboration tools have become essential systems, with new demands placed on networks and data center infrastructure. In this situation, virtual assistants are playing vital roles to share the load from human agents and subject matter experts so that they can focus on real business tasks in hand. As a result, the usage of IBM Watson Assistant to implement these virtual assistants is increased by 60%. The Dialog skill and Search skills of the IBM Watson Assistant are essential components in these implementations. The next couple of sections of this article will provide you with information on how the Search skill of Watson Assistant is enabling virtual assistants or bots to rise up to the next level of human interaction.

What is Search Skill?

When Watson Assistant doesn’t have an explicit solution to a problem, it routes the user’s question to a search skill to find an answer from across your disparate sources of self-service content. The search skill interacts with the IBM Watson™ Discovery service to extract this information from a configured data collection. For example, a remote worker is asking questions such as, “How to setup a VPN?” or, “What are the steps to access a shared network drive?” Answers to these questions sometimes require a document that describes a number of steps.

The following diagram illustrates another example to show end-to-end integration flow of search skill between Watson Assistant and Watson Discovery:

End-to-End Integration Flow for

When you need Search Skill?

• Dynamic situation: If you want to take your bot or virtual assistant to the next level as close to human interaction then you need to use search skill to handle the dynamic situation like Covid-19, where information is changing very fast based on new facts. Search skill can help you in this situation by fetching the document crawl from the website.
• For Longtail Queries: If you want to present the response as a full document or paragraph within it. For example, the query What are the benefits available for full-time employees? Response for this query requires a full document or paragraph related to employee benefits.
• Low Confidence: If your virtual assistant or bot has low confidence in a specific query to respond, it can present a number of documents from the knowledge base.

How to Trigger Search Skill?

Search Skill can be triggered in the following ways:

• From Anything Else Node: Searches an external data source for a relevant answer when none of the dialog nodes can address the user’s query.
• From Dialog Node: If you add a search skill response type to a dialog node, then your bot retrieves documents or a passage from an external data source as the response to a particular question. This approach is useful if you want to narrow down a user query before you trigger a search.
• For Low Confidence Score: Search skills can also be triggered based on the confidence score. Add a folder to the dialog. Position the folder above the first dialog node that you want to de-emphasize. Add the following condition to the folder: intents[0].confidence < 0.5
• Search Skill Only: If only a search skill is linked to an assistant, and no dialog skill is present, then a search query is sent to the Discovery service when any user input is received from one of the assistant’s integration channels.

How to Configure Search Skills?

For configuring Search skill, you can follow the detailed video to configure search skill:

How to optimize the Search?

Most of the time, discovery service is returning the desirable documents to the search skill. However, there are ways you can improve or optimize the search results in Watson Discovery Services:

• Search settings:There are two techniques you can use to optimize search: 1. Query Expansion: You can expand the scope of a query beyond exact matches — for example, you can expand a query for “Covid-19” to include “Coronavirus” and “Novelcoronavirus “ — by uploading a list of query expansion terms. Query expansion terms are usually synonyms, antonyms, or typical misspellings for common terms. For more information, see this link: Query Expansion in Discovery 2. Defining Stopwords: Stopwords are words that are filtered out of queries because they add little value, for example: a, an, the. Adding common words to a stopwords list can also improve the relevance of results for natural language queries. Discovery applies a default list of stopwords for several languages at query time. However, you can define and upload a custom list of stopwords that override the default list. For more information, see this link: Defining Stopwords
• Smart Document Understanding [SDU]: Sometimes, you want to show an appropriate section in the document as a response to the search query. By SDU, you can control the nuance in the document structure. This feature is pretty useful while using search skill. For more information, see the link: Smart Document Understanding
• Relevancy Training: If nothing works and you still need improvement in search results, then try relevancy training. Relevancy training is optional; if the results of your queries meet your needs, no further training is necessary. The relevance of natural language query results can be improved in IBM Watson™ Discovery with training. You can train your private collections using either the Discovery tooling or the Discovery APIs. For more information, see the link to apply relevancy training in your collection: Relevancy Training for Search Results