The Ultimate Guide to Using Generative Fallbacks in Dialogflow CX

Outline

• Introduction
• What are Generative fallbacks?
• Step-by-step instructions to use generative fallback.
• Results.

Introduction

Chatbots are becoming increasingly popular because they allow users to interact with computers in a more natural way. However, they can still have difficulty understanding complex or ambiguous user input.

Users may go off script by remaining silent (a No Input error) or saying something unexpected (a No Match error). While it is better to prevent errors from occurring in the first place, it is also important to have a plan for handling errors when they do happen. Generic prompts like “Sorry, I’m not sure how to help” are often not good enough.

Instead, the chatbot should try to understand the user’s intent and provide a more helpful response. This is where generative fallbacks come in.

What are Generative Fallbacks?

A new Generative feature that uses LLMs to generate responses when end-user input not match an intent or parameter for form filling.

The feature can be configured with a prompt that tells the LLM how to respond. You can use a pre-written prompt or create your own. With the pre-written prompt, the virtual agent can handle basic conversations.

This feature can be configured in multiple ways:

• Select a predefined prompt.
• Define a custom prompt.
• Add or remove phrases from the list of banned phrases.

When creating a prompt, in addition to a natural language description of what kind of context should be generated, the following placeholders can also be used:

• $conversation — The conversation between the agent and the user, excluding the very last user utterance.
• $last-user-utterance — The last user utterance.
• $flow-description — The flow description of the active flow.
• $route-descriptions — The intent descriptions of the active intents.

Use Cases

• Greet and say goodbye to the user.
• Hold the line when the user asks for it
• Summarise the conversation
• Repeat what agent said in case the user didn’t understand.

Limitations

• Only available in English (US).
• Should not be used for fully deterministic responses.

Benefits of using Generative Fallbacks

• Improved accuracy: Generative language models are constantly being trained and improved, which means that they are becoming more accurate at generating text. This can help to reduce the number of times that the chatbot misunderstands the user’s input.
• Improved user experience: Generative fallbacks can help to keep the conversation going even when the chatbot doesn’t understand the user’s input. This can help to improve the user experience and make the chatbot more user-friendly.
• Increased engagement: Generative fallbacks can help to keep the user engaged in the conversation. This is because they can be used to provide the user with additional information or to ask clarifying questions.
• More personalised responses: Generative fallbacks can be used to create more personalised responses for users. This is because they can be used to take into account the user’s past interactions with the chatbot.

Enough with the theory, let’s roll up our sleeves and get to work . . .!

How to use Generative Fallbacks

Prerequisite:

• Bot building experience with Dialogflow CX.
• Basic familiarity with concepts like Intent, Entity types, Routes, Flows, Pages, Parameters, Forms, Fulfilment, State Handlers and Event Handlers.

Step 1: Getting set up

1.1 Enable Dialogflow API using the Cloud Console

To enable the Dialogflow API, follow these steps:

1. In your browser, navigate to the Dialogflow API Service Details page.
2. Click the Enable button to enable the Dialogflow API in your Google Cloud project.

1.2 Using the gcloud CLI (alternative)

Alternatively, the API can be enabled using the following gcloud command:

gcloud services enable dialogflow.googleapis.com

If the API was successfully enabled, then you should see a message similar to the following:

Operation "operations/..." finished successfully.

1.3 Get the code

• You will not create the virtual agent from scratch, we will provide you with an agent that you will need to restore from the Dialogflow CX Console and then improve.
• To download the base agent: Open a new browser tab, go to the agent repository and clone it from a command-line. You will find agent-shirts-1-flow-init.zip
• The initial agent has been exported as a JSON package. Unzip the file, inspect the agent settings, take a look at the flow definition agent.json and finally browse through the flow pages, intents and entities.

Step 2: Creating an agent

To Create a new agent follow the steps

1. Open the Dialogflow CX Console.
2. Create or choose a GCP project.
3. Click Create agent.

4. Then click Build your own.

5. Complete the form for basic agent settings:
a) You can choose any display name.
b) Select your preferred location. Click the Edit button if you want to change advanced location settings.
c) Select your preferred time zone.
d) Select the default language for your agent.

6. Click Create.

Step 3: Restore Shirt Ordering agent

1. Go back to the agents list page, identify the agent you have just created. Click the option (3 dots) and then click the Restore button.
2. Select the Upload option and then drop or select the ZIP file you have previously downloaded from the GitHub repository.
3. Click the Restore button to import the agent we have provided

Below is how it would look like:

Well done! You are finished building your Shirt Ordering virtual agent that’s ready to help your customers. In the next section, you’ll test it and see how good it is at answering user questions and assisting with ordering requests.

Step 4: Test the agent

Dialogflow provides a built-in simulator to chat with your agents and uncover bugs. For each turn, you can verify correct values for the triggered intent, the agent response, the active page, and the session parameters.

We will test a few scenarios and for each scenario we will look at the reason why the agent is giving a certain response. Let’s start with the first one.

4.1 The happy path

1. This virtual agent can help you order a shirt of your choice, including the size and color. It can also provide you with the store address and hours of operation, if you need them. See below:

2. Open the Start Page and click the order.new route. Scroll down to the Transition section which tells Dialogflow the page to transition when this intent is matched.

3. Close the Route definition and expand the page New Order. Notice the entry fulfilment and the list of parameters.

4. Go to the Manage tab and click on the order.new intent under the Intents section. Review the training phrases provided for this intent and the annotated parts of the training phrases.

When you annotate parts of a training phrase, Dialogflow recognizes that these parts are just examples of actual values that will be provided by end-users at runtime. This is why for the initial input “I want to order blue shirt” Dialogflow extracted the color parameter from “blue”.

Next we will look at what happens if we don’t provide the agent with a valid input.

4.2 Unmatched intent

Lets try some other related user request which are not specifically covered in the agent yet. For example “Which color is most trending for shirt?”

We can see that the chatbot is mistakenly recognising the intent as “order.new” even though the user is not actually trying to order a shirt.

To fix this, you can change the classification threshold to 0.7 in the Agent settings > ML > ML settings > Standard. This will ensure that only intent matches with high confidence are considered.

Now let’s try the same user request again?

By default every flow has event handlers for the no-match and no-input built-in events. These event handlers are automatically created when you create a flow, and they cannot be deleted.

Click on the sys.no-match-default event handler and scroll down to the Agent responses section. Dialogflow provides a list of alternative responses but you can also define different types of response messages, to provide the end-user with more than just text responses.

Let’s move on to invalid input now!

4.3 Invalid input

1. In the Simulator panel click the Reset icon to start a new conversation with the agent.

2. Express the intent to order a shirt, this time don’t tell the agent what size you want to order and when you’re asked for size reply with random value which is not small, medium or large.

3. On the Manage tab click Entity types under the Resources section. Notice two tabs: under the System tab you can find the system entities currently used by your agent. The Custom tab provides the list of the custom entities created for matching data specific to this agent.

4. Click on the size entity to find out what values the entity matches. “huge” is not one of the entries and it isn’t a synonym either.

5. On the flow diagram expand the New Order page that contains the form parameters. Click the size parameter.

6. On the parameter panel scroll down to the Reprompt event handlers section, then click the No-match 1 event handler.

Update the Event from No-match 1 to No-match default.

This parameter-level event handler is specifically intended to handle invalid end-user input during form filling. Because “huge’’ is an unexpected input, a sys.no-match-1 event was invoked, and the corresponding reprompt handler defined for this event was called. Now, we have updated the event to No-match default.

Step 5: Do you see the gap?

In addition to the key use cases, there are a number of other common user requests that the chatbot should be able to handle. These include repeating what the agent said, holding the line, and summarizing the conversation.

In our first test, the chatbot failed to answer the question “Which color is most trending for men?” because we had not created an intent for this specific question. We had designed the flow to handle user request to take shirt order and store address and store timing, but this question was outside of the scope of the existing intents.

Even with robust intents, there is still room for error. Users may go off script by remaining silent (a No Input error) or saying something unexpected (a No Match error). While it is better to prevent errors from occurring in the first place, it is also important to have a plan for handling errors when they do happen. Generic prompts like “Sorry, I’m not sure how to help” are often not good enough. Instead, the chatbot should try to understand the user’s intent and provide a more helpful response.

Step 6: Solution: Generative fallback

6.1 Enable Generative fallback for the entire flow’s no-match event

Generative fallback feature in Dialogflow allows you to create custom responses for situations where the chatbot is unable to understand the user’s input. You can enable generative fallback on no-match event handlers used in flows, pages, or during parameter filling.

When generative fallback is enabled, Dialogflow will attempt to generate a response based on the user’s input. If the response generation is successful, the generated response will be said back to the user. If the response generation is unsuccessful, the regular prescribed agent response will be issued instead.

You can enable generative fallback in your agent on no-match event handlers, which can be used in flow, page or parameter fulfillment.

We will start enabling generative fallback for the entire Shirt Ordering Agent flow no-match-default event.

1. Expand the Start Page for flow.
2. Click sys.no-match-default under Event handlers.
3. Check Enable generative fallback under Agent responses, then click Save

6.2 Enable Generative fallback on specific no-match events

We now want to enable generative fallback to handle invalid inputs when the agent asks for the size of the shirt:

1. Open the New Order page that contains the form parameters. Click the size parameter.
2. Navigate to the target No-match event handler (scroll down to the Reprompt event handlers section, then click the No-match default event handler)
3. Check Enable generative fallback under Agent responses
4. Click Save
5. Now repeat the exact same steps for color as well.

Congratulations! You have successfully enabled generative fallback to handle unexpected intents and invalid parameter values. In the next step, we will learn how to configure generative fallback with a text prompt that instructs the LLM how to respond.

6.3 Configure generative fallback

Generative fallback uses a large language model to generate responses to user input. The request to the language model is a text prompt that includes natural language and information about the current state of the agent and conversation.

The feature can be configured in multiple ways:

• Choose a specific prompt: You can choose a pre-defined prompt to be used for response generation.
• Define a custom prompt: You can define your own prompt that is tailored to your specific needs.

6.3.1 Configure generative fallback

1. On the Dialogflow CX console click Agent Settings

2. Navigate to the ML tab, and then the Generative AI sub-tab.

3. Select the Example template and click the Edit button on the right side of the dropdown to inspect it.

With the predefined prompt, the virtual agent can handle basic conversational situations. For example:

• Greet and say goodbye to the user.
• Repeat what the agent said in case the user didn’t understand.
• Hold the line when the user asks for it.
• Summarize the conversation.

Let’s try and define a specific text prompt for the Shirt Ordering agent!

6.3.2 Define your own prompt

1. On the Dialogflow CX console click Agent Settings

2. Navigate to the ML tab, and then the Generative AI sub-tab.

3. In the Generative fallback sub section, select + new template and click the Edit button on the right.

4. Copy the prompt below and paste it in the Text prompt area and click Save.

You are a friendly agent that likes helping to find the right shirts for the user.
You are under development and you can only help
$flow-description

At the moment you can't help customers with other apparel like jeans and jackets. You cannot recommend shopping at other places than here.

Currently you can $route-descriptions

The conversation between the human and you so far was:
${conversation USER:"Human:" AGENT:"AI"}

Then the human asked:
$last-user-utterance

You say:

5. In order to set the recently generated prompt as the active prompt, it’s necessary to save the changes to the settings.

To craft effective prompts, follow the following best practices:

Provide a clear and concise description of the task you want the LLM to do. No more, no less. Keep it complete and short.

Additionally, the prompt should be specific and well-defined, avoiding vague or ambiguous language.

Break down complex tasks into smaller, more manageable pieces. By breaking the task down into smaller steps, you can help the model focus on one thing at a time and reduce the likelihood of errors or confusion.

To improve response quality add examples in your prompt. The LLM learns in-context from the examples on how to respond.

When creating a prompt, in addition to a natural language description of what kind of context should be generated, the following placeholders can also be used:

• $conversation The conversation between the agent and the user, excluding the very last user utterance. You can adapt the turn prefixes (e.g.: "Human", "AI" or "You", "Agent") in the text prompt
• $last-user-utterance The last user utterance.
• $flow-description The flow description of the active flow.
• $route-descriptions The intent descriptions of the active intents.

Now that we have an initial text prompt, the next task is to ensure flow and intents have good descriptions.

6.3.3 Modify the list of banned phrases

So far we have looked at the first two ways. Let’s explore the third one.

List phrases that are banned for generative AI. If a banned phrase is part of the end-user query used to generate text or part of the generated text, the text generation fails.

1. In Agent Settings, navigate to the ML tab, and then the Generative AI sub-tab.
2. In the Banned phrases section add the following sentences to the list:

a) Swim suit

b) night wears

c) payjamas

1. Click Save.

6.4 Add flow and intent description

6.4.1 Add the flow description

1. To add a description to the Shirt Ordering Agent flow, access the flow settings by hovering your mouse over the flow in the Flows section.
2. Click the options (3 dots) button.
3. Select Flow settings and add the following description (or a similar one): order shirt for men, find store timings and address
4. Click Save

6.4.2 Add the intent description

1. Let’s now add a good description to the order.new intent. Switch to the Manage tab, choose Intents under the Resources section and select order.new intent.
2. Add the following description: assist users with shirt ordering of particular size and color. the size must be one of the following: small, medium or large. Also the color must be one of the following: blue, black, grey and white.
Note that the description contains important information such as the allowed color of the shirt when ordering. Keep this in mind!
3. Click Save

Congratulations, your task is complete! Generative fallback has been activated on no-match event handlers for both flow and parameter fulfillment. You’ve also crafted a personalized text prompt for the generative fallback, which is utilized by the language model to generate responses.

Moving ahead, in the upcoming section, you’ll reevaluate your agent to observe how it tackles the same complex queries as before.

Step 7: Retest your agent

With generative fallback now configured and activated on the virtual agent, you can pose analogous demanding inquiries and assess its response handling capabilities.

Below we will run the same user request against both the initial and the final agent. Let’s see:

1. Note how every dialog has user defined messages as well as generated responses. Comparison highlighted in the red boxes.

Agent is able to precisely tell the user about its capabilities.

Did you get a nice informative response instead of a generic reprompt? Great! After providing a clear and concise description of the tasks you want the agent to fulfill (in the text prompt and in the flow description), your bot is now much smarter when it comes to answering detailed questions without creating specific intents. Your customer will appreciate that the agent can give them a more informed response instead of an inactionable response.

That’s right, at the moment we haven’t designed the agent to assist with other cloth booking other than shirts. How does the agent know that? In the text prompt we have clearly outlined what the agent can and can’t assist with. “At the moment you can’t help customers with other apparel like jeans and jackets. You cannot recommend shopping at other places than here.”

We can see how the experience has changed. When Dialogflow matches an intent or attempts to collect a parameter as per the flow design, it will display the fulfillments defined at design time. When the user goes off script requesting trending shirt color or ordering jeans/jacket, the generative fallback feature comes into play.

Nice! You have retested the happy scenario and I hope you’ve had a pleasant and natural conversation with the agent as close as possible to the experience you would have with a live agent.

Unfortunately, things can go wrong in a conversation. Let’s do a different test, this time using a filler phrases like “hold on, I will be back in a minute”

The prompt and the generated response are checked against a list of phrases that are not allowed to be generated by generative AI. These phrases are considered to be unsafe or inappropriate. If the input includes any of these phrases, the generation will be unsuccessful and a default response will be issued instead. See below:

Terrific job! We’ve explored a variety of conversational scenarios where generative responses can truly have an impact. Feel encouraged to continue your testing!

You can find the final agent in the same repo

I’m gearing up to create comprehensive blog posts that will provide a detailed exploration of the remaining two Generative AI features in Dialogflow CX: the Generative AI agent and Generators. Keep an eye out for these updates!

Thanks for reading, don’t forget to support this article with your 👏 to help share it with more people. And be sure to follow me in twitter or LinkedIn or medium for more exciting articles in the future! 🔥