Guide to configure Vertex AI Search as a Webhook in Dialogflow CX

Unlock the Power of Dialogflow CX Datastores with Webhooks

You’ve likely heard about Dialogflow CX’s Datastore agents — a fantastic way to generate AI-powered responses using your website, documents, and even BigQuery data. Instead of building complex conversation flows, simply feed in your data, configure the datastore, and boom! Your agent can chat.

But what if you crave extra control over when and how your datastore gets used in conversations? That’s where webhooks shine.

Ready to dive in? In this article, we’ll break down the why’s of Datastore Webhooks and walk you through a detailed setup guide.

PS: I might have used Datastore and Vertex Search interchangeably in this article

When to use this implementation

Dialogflow CX datastore Constraints: When Setup Within a Page Event Handler

It’s important to understand the constraints of Datastore knowledge handler when configured this way:

• User Utterance Reliance: Datastore only reacts to direct user text input on the associated page. It won’t trigger based on internal parameter values or other routing logic.
• Route Priority: If any regular Dialogflow CX route matches the user’s input, Datastore won’t be triggered, even if the route leads to a dead end.
• Custom Invocation Missing: You cannot proactively call Datastore from other parts of your flow, or customize the conditions that trigger its use.

When to Choose Vertex Search via Webhook: Key Scenarios

Outgrowing the built-in Datastore functionality in Dialogflow CX? Using Vertex Search with webhooks gives you the power to:

• Intelligently Route Queries: Got multiple datastores? Base search routing on NLU analysis of the user’s intent, directing them to the most relevant page or flow associated with a datastore.
• Handle Datastore Duplication: Have multiple datastores even for the same kind of data (e.g., URLs, FAQs)? Create separate pages to leverage each one individually, unlocking more targeted search results.
• Integrate with Existing Chatbot Logic: Need to use pre-existing user input (stored in session parameters) as your search query? Webhooks let you seamlessly pull this data, even if it comes from a guiding playbook.
• Conditional Search: Trigger Vertex Search only when specific conditions are met within your conversation flow. This means precise control and avoids unnecessary searches.
• Customize with Parameters: Want to refine search results by passing additional user-provided parameters alongside their query? Webhooks make it possible

Caveats

Remember, this enhanced setup offers flexibility but also comes with trade-offs:

• Response Generation is DIY: Unlike traditional Infobot, you’ll be responsible for handling the raw datastore output. This means using Cloud Functions and potentially Dialogflow Generators to craft the final, user-friendly response. More work, but also more control!
• Think About Speed: Introducing this chain (DFCX -> Cloud Functions -> Cloud Run -> Cloud Functions -> DFCX) can add some latency to your chatbot’s responses. It’s crucial to optimize your processes and potentially consider caching results for time-sensitive scenarios.

Architecture

Pre requisite

• Dialogflow CX Foundations: Experience building chatbots with Dialogflow CX, including a solid understanding of flows, pages, and intents,entities.
• GCP Datastores: Knowledge of creating and configuring Search and Conversation datastores within the Google Cloud Platform.
• Cloud Functions Basics: Comfort writing and deploying Cloud Functions
• Python : Knowledge in Python for make api requests, data manipulation and handling web requests.
• GCP Foundations: Some familiarity with how GCP services interact will make the integration process smoother.

So let us finally jump into the implementation step after all the boring theory

Use Case: Your Pixel Expert Chatbot

Imagine a chatbot that’s a walking encyclopedia of Pixel 6 and Pixel 7 knowledge. Let’s build a flexible agent that can:

• Friendly Welcome: Greet the user warmly and get started.
• Pixel Power: Prompt the user whether they’re interested in Pixel 6 or Pixel 7 specs and features.
• NLU power: Identify the intent(in our case we have defined an intent “pixel) and do the routing. Before routing, Also do the model param detection based on the pixel training phrases annotation
• Reprompt: If the model name is not specified in the query prompt user to give the model name
• Tap into Datastore Knowledge: Behind the scenes, search a powerful datastore that’s filtered specifically for the user’s Pixel model, providing the most relevant answers.

Why do we need this implementation of webhook?

1. If we use OOB datastore event handler then we cannot capture the parameter and parse it to datastore within dialogflow cx. By also sending this parameter it improves the accuracy of data store retrieval. With OOB datastore you can send parameters to datastore only from client side detect intent api request
2. With OOB datastore event handler, we cannot define intents. If we use datastore as an event handler then we cannot do intent detection as that datastore event handler works when there is a No Match. In some cases we might need to do the intent and handle the datastore calls differently for different routes
3. We are able to rephrase the user query using LLMs a.k.a generators so that retrieval accuracy can be improved.
4. We can also increase the Cloud Function resources to improve the latency of response. But in case of OOB datastore event hanlder we dont have that flexibility

Ready to see how it’s done? Let’s dive into the setup…Steps to complete the set upSteps to complete the set up

Steps to complete the set up

APIs to enable

You can ensure that the APIs are enabled through the Google Cloud Console UI. A green checkmark indicates that the API is enabled:

• Cloud Build API
• Cloud Functions API
• Cloud Logging API
• Cloud Pub/Sub API
• Cloud Run API
• Dialogflow API
• Discovery Engine API

Enable APIs

APIs can also be enabled using CLI .Example :

gcloud services enable dialogflow.googleapis.com

Upload files to Cloud Storage

Okay, you’ve got Wikipedia PDFs for Pixel 6 and Pixel 7 ready to go. Here’s how to get them into Cloud Storage and set up a schema for laser-focused search:

1. Upload to Cloud Storage: Transfer your PDF files to a Cloud Storage bucket. This is where they’ll live for access by Vertex Search.
2. Craft a JSONL Schema: Define a JSONL file that outlines the structure of your PDF documents (e.g., fields like “title”, “model keys”, “gcs urls”).

Why a Schema Matters

• Speed Boost: When you provide search parameters that match your schema, Vertex Search can zoom in on relevant content much faster.
• Accuracy FTW: A schema lets us differentiate between the two Pixel models. If a user’s query is vague, we’re more likely to pull the correct info because we know how the documents are organized.

STEPS

1. Create a bucket based on the instruction given here in the same project where we will have our agent
2. Create a folder called pixel in that bucket
3. Inside that pixel folder upload 3 files(2 pdf,1jsonl) present in this git location
4. Ensure that you update the jsonl file to replace your drive name instead of mine.

Set up Datastore

In this section we will see how to create a datastore out of unstructured docs such as PDFs we have uploaded in the previous section.

STEPS

1. Go to GCP console and open the project where you created/will create the agent and search for “Vertex AI Search and Conversation” and go to that service
2. Click on New App and select Search app type

3. Give a name and company Name and keep default value in region and click continue

4. In the next screen click on CREATE NEW DATA STORE and select Cloud Storage

5. In the next screen, select the folder Pixel you created in previous section in Cloud Storage bucket and select JSONL for unstructured documents with metadata in the radio button section and click CONTINUE.

6. Enter Datastore name in the next screen

7. In the next screen , you will see the datastore name we just created in step 6. Just select it and click the create button at the bottom.

8. Once created you will see something like below and make sure to note down this datastore ID which will be used in Cloud Functions.

9. In the SCHEMA tab, you will see that the schema is created based on the uploaded JSONL file.

Set up Cloud Function

STEPS

1. Go to GCP console
2. Search and go to Cloud functions
3. Click on CREATE FUNCTION

4. Fill details as below and expand “Runtime, build,connections and security settings” section in the same page at the bottom

5. In the same screen scroll down to the end in “Runtime, build,connections and security settings” and select the existing service account “discovery engine” and click Next.

6. Next select Runtime as python 3.12 and copy paste the code from this git into main.py and requirements.txt.

7. Before clicking DEPLOY make sure to update your datastore id and location obtained in the previous section of creating the datastore and projectid you are using.

8. Once the build is complete , copy the cloud function url generated and displayed in the same screen which will be used in the Dialogflow CX agent.

Set up the Agent

Do not worry. The agent is already created for you. Just download the agent from here and restore it in your own project from dialogflow cx console. To complete the set up follow below steps.

Create Agent

1. Open dialogflow cx console
2. choose a GCP project where you have executed above sections
3. Click on create agent and follow the steps here to create an empty agent. Make sure you selected “build your own” in step 4 in the above link and agent type= CX flows in step 5 in above link

Restore agent

1. Download the agent zip from here.
2. Open dialogflow cx console and select the project where you have created the agent in previous section
3. Click on 3 dots beside the agent you create in previous section
4. Click Restore button
5. Browse and select the agent zip file downloaded in step 1.
6. Once the agent is restored , go to Manage -> Webhooks -> Open datastore_webhook created already and input the cloud function URL we got in Cloud function set up section

IAM role for agent’s service account

Now we need to give the IAM role to this agent’s service account to hit Cloud Function.

STEPS

1. Go to Agent settings -> Share -> find entry with Role = Service Agent and Email type = Service Account.

2. Go to Cloud Console and Go to IAM.

3. Click on Grant Access and enter this this service account’s id and Click Add role and grant Cloud Functions Invoker and Cloud Run Invoker roles. Now you are all set.

Things to note in this set up

In Agent:

1. This agent and Cloud function is not optimized for latency. To improve latency , increase the resources to Cloud function
2. We are using generators to summarize the user query before hitting the Cloud Function.
3. We send the model and user query as input to the Cloud Function and it sends back certain fields and we are capturing just extractive_answers,summary field which we are displaying back to the user and link of the doc.

4. We have used a conditional response like this to display back the message

In Cloud function:

1. We send these inputs(query and model) to CF and it hits the datastore we created and get us back the response

    request = dengine.SearchRequest(
        serving_config=serving_config,
        query=data['query'],
        query_expansion_spec={ "condition": "AUTO" },
        spell_correction_spec={ "mode": "AUTO" },
        filter=f"model: ANY(\"{model}\")",
        content_search_spec=content_search_spec
        )

In Datastore:

1. In the datastore we created in Search and Conversation service in GCP, you will see the schema tab in that datastore. Schema is automatically detected by the JSONL file we uploaded in the Cloud Storage along with pixel pdfs.

1. So when Dialogflow cx sends query and MODEL on request, datastore is going to search the mapping in its SCHEMA with the MODEL field and based on the mapping it is going to open the respective pdf for response retrieval for the given query.

Testing

Now that we are all set, let us play around it. Please note the parameter section in the left side. Model/query is captured by dialogflow cx from the user utterance and extractive_answers, summary and title is set by datastore response via webhook.

Test 1:

In below test , we see user has asked “give me pixel 6 spec”

Here’s how the magic happens:

• Understanding the User: First, the bot analyzes the user’s words and determines they’re asking about a Pixel phone (that’s NLU in action!).
• Helpful Routing: The bot pinpoints the “Pixel Intent” we’ve set up, ensuring the conversation is on the right track.
• Parameter Power: It cleverly extracts the specific Pixel model the user mentioned, customizing the search that’s about to happen.
• Generator + Webhook Combo: Generators process the user’s query, and behind the scenes, a webhook triggers a search of our datastore.
• The Final Step: The bot neatly presents the results to the user.

Test 2:

In the test below , we see the user has asked “when was pixel released” . In this case the NLU identified pixel intent but did not capture the model entity as it is not present in the query. So it prompted again to the user to say it and then given the response back.

In this agent we have used generators ie LLMs to capture the user query in a variable. If you want to avoid LLM calls you can go with below approach

Approach 2

We used Generators(LLMs) to capture user query. But instead of using LLMs to capture the query and avoid llm calls, we can collect model parameter in step 1 and collect user query in step 2 and hit the webhook to get the datastore

To use this approach , use the agent file from this location instead to restore and do rest all steps as is as defined above