Enhancing Conversational AI with Dialogflow and Vertex AI: A Deep Dive into Vertex AI Agent Builder

Author: Syed Ali Naqi at DreamAI Software

In today’s interconnected digital environment, effective engagement with users is critical for enterprises. This blog explores the transformative potential of Vertex AI Agent Builder, a sophisticated tool from Google Cloud that empowers developers to create dynamic conversational agents, even without extensive machine learning expertise.

Challenges in Building Conversational Agents:

Building efficient conversational agents involves addressing multiple complexities:

• Complex Interaction Needs: Agents must manage nuanced, context-rich dialogues that feel natural to users.
• Scalability and Reliability: These systems need to handle variable user loads consistently and reliably.
• Integration of Diverse Data Sources: Effective agents draw on a mix of structured and unstructured data sources.

Introducing Vertex AI Agent Builder:

Vertex AI Agent Builder leverages Google’s advanced foundation models and search technologies, including the robust capabilities of Dialogflow and Vertex AI, to simplify the creation of enterprise-grade conversational agents.

Key Features:

• Rapid Deployment: Developers can quickly set up sophisticated agents.
• Utilization of Dialogflow: Integrates with Dialogflow to enhance conversational capabilities, allowing the agents to process and understand natural language effectively.
• Data Store Agents: Automatically generate agents capable of intelligently discussing content from provided documents or websites.

Overview:

In this blog, we will explore how to build two types of AI-powered chatbots using Vertex AI Agent Builder.

First, a virtual agent designed to assist customers with inquiries about products and devices in the Google Store, such as phones, watches, laptops, and smart home devices. You will learn:

• How to create a Data Store Agent.
• Add unstructured data and enable voice and chat experiences.
• Test and simulate customer interactions.
• View conversation history and analytics.

Second, we will develop a virtual agent to aid individuals looking to donate blood, ensuring they meet the necessary eligibility requirements. This agent will utilize real public data and Google’s generative LLMs in Dialogflow CX fulfillment. You will learn:

• The use of generative fallback, which will enable the agent to provide helpful responses even when user queries do not match a predefined intent.
• Use knowledge handlers and configure generator text prompts.
• Testing and simulating complex customer queries that trigger generated responses.

Step-by-Step Guide: Building Virtual Agents

Before creating virtual agents, make sure the Dialogflow and Vertex AI Search and Conversation APIs are enabled.

To enable the Dialogflow API, follow these steps:

1. In your browser, navigate to the Dialogflow API Service Details page.
2. Click the Enable button to enable the Dialogflow API in your Google Cloud project.

To enable the Vertex AI Search and Conversation API, follow these steps:

1. First, log in to your Google Cloud console and go to the Agent Builder page.
2. Read and agree to the Terms of Service, then click Continue and activate the API.

Building a Virtual Agent for the Google Store

First, we will create a new chat app for our virtual agent and configure it with a data store. The purpose of this agent is to assist customers who have questions about products in the Google Store.

We’ll use the App Builder console and Dialogflow CX console to create, configure, and deploy our virtual agent that can handle questions and answers using a Data Store Agent.

1. To create a new chat app, you can either:

• Navigate to the Google Cloud console, go to the Agent Builder page and click +New app button.

• Or navigate to the Dialogflow CX console, where it will show you the list of GCP projects that you have already created. Once you select your project, it will take you to the agent creation page.

• Click Create agent button and select the Auto-generate option, then you’ll be redirected to the Agent Builder app creation page.

2. In the Select app type pane, select Chat as the type of app that you want to create.

3. In the Company name field, input a company name (e.g., Google Store). This parameter is used to define the company that your agent represents and the scope of your agent.

4. Also specify an Agent name of Google Store.

5. Select the global (Global) as the location for your app, and then click Continue.

6. After you click continue, it will take you to the datastore page where you need to create a datastore.

7. In the Data stores pane, click Create new data store.

8. In the Select a data source pane, select Cloud Storage as the data source for your data store.

9. In the Import data from Cloud Storage pane, make sure Folder is selected.

10. In the gs:// field, enter the following value (excluding the prefix ‘gs://’):

cloud-samples-data/dialogflow-cx/google-store

This folder contains sample data for this tutorial, and note that the gs:// prefix is not required.

11. Select Unstructured documents as the type of data you are importing and then click Continue.

12. In the Configure your data store pane, enter a name for your data store (e.g., in our case, we have entered Google Store), and then click Create.

13. On the Data stores page, select your new data store (i.e., Google Store) and click Create to create your chat app.

Note: It can take up to 4 hours for your documents to be available and ready for use by your agent while your newly added data is being indexed.

14. On the Data page of your datastore, navigate to the Activity tab to monitor the status of your data ingestion. The Status column will display “Import completed” once the import process has concluded.

15. To view the number of documents imported, click the Documents tab.

Test your Virtual Agent

Testing is a critical component of developing conversational agents. It allows developers to identify bugs, understand limitations, and simulate real customer interactions. To ensure your chatbot performs as intended, initiate an interactive session and engage it with diverse questions a customer might pose. This process helps refine the agent’s responses, ensuring they are both accurate and contextually appropriate.

To test your virtual agent, follow the steps below:

1. From the Agent Builder console, click the name of your chat app (which is Google Store in our case), and it will redirect you to the Dialogflow CX console for further testing and customization.
2. In the Dialogflow CX console and from within your agent, click Test Agent to open the Simulator.

3. Now you can interact with your agent by asking questions in Talk to agent field. Ask the agent some questions about various products such as:

• How long does the battery in the Pixel 7 Pro last?
• Is the Pixel Watch water resistant?
• Can I display my Google Photos on a Nest Hub?

Ask other questions about other products and devices in the Google Store.

Note: If your agent is giving you responses from the default negative intent (e.g., “I’m sorry, I didn’t get that. Can you rephrase your question?”), then be aware that it can take up to 4 hours for the agent to be ready for use while your domains, URLs, or documents are being indexed.

Enable Voice Calls

Once you are satisfied with the performance and functionality of your agent after thorough testing, the next step is to enhance its accessibility. By integrating a phone gateway into your bot, you leverage Google Cloud’s advanced Speech-to-Text and Text-to-Speech capabilities. This integration allows your virtual agent to seamlessly interact with users through voice calls, broadening the scope of your customer service solutions.

Follow the steps below in order to integrate the phone gateway into your chat bot:

1. In the Dialogflow CX console and from within your agent, click the Manage tab in the sidebar, then go to the INTEGRATIONS section and click Integrations.
2. Click Manage under the CX Phone Gateway section at the top of the Integrations page.

3. On the Phone numbers page, click Create New button to start the process of creating a new phone number for your virtual agent.

4. Now select a Country code (e.g., +1 (United State)) and click Request button to continue to the next step. Area code field is optional so we are leaving it empty, but you can provide the area code as well if you want.

5. After clicking the Request button, you will be directed to a page where you can select a phone number. Select one of the phone numbers that appears, input a Display name (e.g., Google Store Agent), then click Save.

Now your agent is ready and you have given your virtual agent its own phone number and voice. Call your agent and ask it some questions like:

• Do the Pixel 7 Pro and Pixel 7 both have face unlock?
• What coverage does Preferred Care provide for a Pixelbook Go laptop?
• Do the Pixel Buds Pro have active noise cancellation?

Further explore your agent’s capabilities by asking questions about the various products and devices available in the Google Store. This exercise allows you to evaluate how effectively your agent retrieves and provides information across a broader range of inquiries.

Embed Chat Widget on Website

With your bot now equipped with a phone gateway for voice interactions, the next step is to expand its accessibility by embedding a chat widget on your website. This addition enables customers to engage with the bot not only through voice calls but also via text-based chat.

In this blog post, we will utilize JSFiddle, an online code editor, to demonstrate the seamless integration of your chatbot into a website, offering a dual-mode communication channel for enhanced customer interaction.

Follow the steps below to embed chat widget on a website:

1. In the Dialogflow CX console and from within your agent, click the Manage tab in the sidebar, then go to the INTEGRATIONS section and click Integrations.
2. Now scroll down a bit and find Dialogflow Messenger. Click Connect under Dialogflow Messenger.

3. When you click Connect, Dialogflow Messenger popup will appear that will show you some options to select from. Select Unauthenticated API (anonymous access) as access type and choose a UI style that you like for your chat widget. Now click Enable the unauthenticated API.

4. Once you enable the unauthenticated API, you will be provided with an HTML code that you can put into your website for your chat widget integration.

5. To test this chat widget, open the online code editor JSFiddle, paste your Dialogflow Messenger HTML code in it under HTML section and hit Run button. Now you can test and chat with your virtual agent via chat widget. Let’s ask some different questions this time like:

• Can you recommend a compatible wireless charger for the Pixel 7 Pro?
• Are there different color options available for the Pixel Buds Pro?
• Can you swim with the Pixel Watch? Is it water-resistant?

As you can see, the testing results are quite promising and your virtual agent can now handle questions and answers from your customers via chat or voice, whichever they prefer, improving their overall experience with your service.

Conversation History and Analytics

When testing your virtual agent, if the responses fall short of expectations, consider expanding the data store by adding more URLs. This enhancement can significantly refine how your bot processes and answers questions. Beyond exploring various customer interaction paths, another effective way to identify issues and friction points is through the use of conversation history and analytics. These tools provide invaluable insights into the agent’s performance, helping you make informed improvements to the user experience.

Let’s see how we can use Dialogflow conversation history and analytics to gain insights about the performance of our virtual agent.

1. In the Dialogflow CX console and from within your agent, click Agent settings in the menu.
2. Under Logging settings, Enable Conversation History and save your new settings.
3. Engage with your bot using the agent simulator, voice integration, or chat integration and ask it some questions.

4. Engage with your bot using the agent simulator, voice integration, or chat integration and ask it some questions.

5. Once your conversation with the agent concludes, navigate to the Dialogflow CX console. Within your agent’s interface, select the Manage tab from the sidebar, then click on Conversation history.

6. Choose a recent conversation to review. Analyze the duration of the conversation, the responses provided by the agent, and identify which intents were matched during the interaction.

7. In the Dialogflow CX console, access your agent and select the Manage tab from the sidebar. Then, click on Analytics. This section provides a comprehensive summary of various statistics, including agent requests and responses, as customers interact with your agent. This data is crucial for monitoring performance and making informed decisions to enhance user interactions.

Excellent! You now possess the knowledge to delve into specific conversations in greater detail and review additional metrics concerning your agent’s responses and customer interactions. This valuable data enables you to evaluate how your agent is performing in a live environment. Utilizing these insights, you can identify which websites and documents to incorporate into your knowledge base, thereby enhancing both your agent’s capabilities and the overall customer experience.

Creating a Virtual Agent for Blood Donation Assistance

Now that you’re familiar with creating a Data Store Agent, adding unstructured data, and enabling voice and chat experiences — not to mention testing and checking out the analytics — it’s time to dive into some of the more advanced features of Dialogflow. Using Google’s generative LLMs in Dialogflow CX, we’ll next develop a virtual agent tailored to help individuals check if they meet the requirements for donating blood, making the process straightforward and informative.

1. Create a new chat app as you have done before for Google Store Agent, but this time, add Save a Life as Company name and under Agent name field, give the following name: Blood Donation Agent.
2. Select the global (Global) as the location for your app, and then click Continue.

3. In the Data stores pane, click Create new data store.

4. In the Select a data source pane, select Cloud Storage as the data source for your data store.

5. In the Import data from Cloud Storage pane, make sure Folder is selected.

6. For blood donation data, in the gs:// field, enter the following value (excluding the prefix ‘gs://’):

cloud-samples-data/dialogflow-cx/arc-lifeblood

This folder contains sample data for this tutorial, and note that the gs:// prefix is not required.

7. Select Unstructured documents as the type of data you are importing and then click Continue.

8. In the Configure your data store pane, enter following name: Australian Red Cross Lifeblood Unstructured as your data store agent, and then click Create.

9. On the Data stores page, select your new data store (i.e., Australian Red Cross Lifeblood Unstructured) and click Create to create your chat app.

Note: It can take up to 4 hours for your documents to be available and ready for use by your agent while your newly added data is being indexed.

Configure the agent to answer blood eligibility FAQs

Follow the steps below to configure the agent to answer blood eligibility FAQs:

1. From the Agent Builder console, click the name of your chat app (i.e., Blood Donation Agent), which will redirect you to the Dialogflow CX console for further testing and customization.
2. In the Dialogflow CX console and from within your agent, click Agent settings (top right corner of the page), then go to the Generative AI tab and lastly open the Data Store tab.

3. Fill out the form as below to generate the following data store prompt: Your name is Donate, and you are a helpful and polite chatbot at Save a life, a fictitious organization. Your task is to assist humans with eligibility information.

4. Switch to the Build tab and open the Start Page.

5. Click the sys.no-match-default event handler. Unless the box is already checked, enable the generative fallback feature.

6. On the Start Page click Edit data stores to look into the Data store settings.

7. The data store you have previously created has been already selected for you by Dialogflow.

8. Scroll down to Agent Responses under Fulfillment. Dialogflow has pre populated Agent says with the parameter $request.knowledge.answers[0] which at run time contains the top answer to the user’s question.

Test the Agent

Once the documents are available and ready for use by your agent, check out how good the answers are.

1. Click Test Agent (top right corner of the page) to open the Simulator.
2. Ask questions that you expect to find in the FAQ page of the website like:

• What age do I need to be to donate?
• Can pregnant women donate?
• I’ve just come back from a trip to Africa. Can I donate?
• How can I schedule an appointment?

Note: If your agent is giving you responses from the default negative intent (e.g., “I’m sorry, I didn’t get that. Can you rephrase your question?”), then be aware that it can take up to 4 hours for the agent to be ready for use while your domains, URLs, or documents are being indexed.

3. Let’s challenge the agent with a question completely unrelated to blood donation, such as:

• User: “What’s the weather like in Melbourne?”
• Agent Response: “I’m sorry, I can only provide information about eligibility to donate blood. How can I help you today?”

As you can see, the answer has AI generated content in it and derives from the text prompt that Dialogflow has created starting from the knowledge connector setting provided before: “Your name is Donate, and you are a helpful and polite chatbot at Save a Life. Your task is to assist humans with eligibility information”. This text prompt contains the company name, the agent name and most importantly what is in its scope which is used by Dialogflow to generate the agent response.

Conclusion

The integration of Vertex AI Agent Builder with Dialogflow offers a robust platform for building conversational agents that can enhance customer engagement through intelligent, context-aware interactions. The blog outlines a detailed step-by-step process for creating virtual agents tailored to specific needs, such as customer support in the Google Store and assistance for blood donation eligibility.

By leveraging Google Cloud’s advanced machine learning technologies and rich data sources, these agents are not only capable of understanding and responding to complex queries but also adapted to deliver personalized user experiences. Through thorough testing, integration with voice capabilities, and embedding on websites, these agents promise to significantly improve how businesses interact with their customers, making these interactions more intuitive and efficient.

References

1. Google Cloud Dialogflow documentation — Comprehensive resource detailing the functionalities and setup procedures for Google Cloud Dialogflow, aiding developers in creating intelligent conversational agents.
2. Vertex AI Agent Builder guide — Provides detailed instructions and insights on using Vertex AI Agent Builder, allowing developers to craft sophisticated AI agents efficiently.
3. Introduction to building conversational AI with Google Cloud — An essential guide for developers looking to understand the foundations and advanced implementations of conversational AI within the Google Cloud ecosystem.

These references provide detailed documentation and further reading that can help anyone delve deeper into the specifics of implementing the technologies discussed in this blog.