Ask your documents anything w/ Vertex AI Search

Published in

happtiq Data & AI Hub

12 min readNov 3, 2023

Introduction

Picture this: you’ve got a treasure trove of data, a goldmine of insights waiting in your documents to be unearthed. But here’s the catch — the traditional path to access and utilize that data is a labyrinth of challenges, from complex setups to deciphering content.

But what if we tell you there’s a way to flip the script?

In this post, we’re about to unveil your secret weapon — a technology that makes harnessing the power of Google’s latest AI as easy as a few clicks, Vertex AI Search.

What is Vertex AI Search (formerly: GenAI App Builder)

Vertex AI Search is one part of Google’s latest released Generative AI products. It offers a low- to no-code platform that enables developers to quickly and easily build and deploy Generative AI-powered semantic search applications that can understand and respond to questions about your own, internal data instead of only relying on general knowledge from traditional search engines.

Vertex AI Search (formerly: GenAI App Builder) as part of Google’s other AI services

Thus, Vertex AI Search dramatically reduces the time and complexity of building up a personalized search experience. This is done by providing access to a variety of pre-trained Generative AI models and the ability to include your own structured and unstructured data sources.

Overall it provides a variety of features, including:

An easy-to-use interface for building and setting up a search application without making it mandatory to have machine learning experience
Pre-trained GenAI models for tasks, such as text summarization, or question answering (Q&A)
Include multiple data sources, including data from your texts, PDFs, websites, BigQuery datasets, or include data from other APIs
Built-in security and compliance features

This allows us to build a wide range of search applications, including:

Internal search engines for employees: Build your internal search engines that can help your employees find the information they need quickly and easily, without worrying about at which of the many locations this information can be found
Product search engines: Build your own product search engines that can recommend products to customers based on their individual needs and preferences
Knowledge Base: Build a knowledge base that can help people learn new things and find answers to their questions

Challenge

One major challenge for organizations nowadays arises through the huge variety and amount of their generated data, its discoverability. As an organization you have a vast amount of data, scattered across many different sources (databases, documents, web pages) which makes it rather difficult to find what you’re looking for, whether it’s about internal or publicly available data.

For this, you need some kind of search engine, tailored to your own data, which can also combine public data. However, this leads to some more challenges, such as

Traditional search experience with keyword search is too restricted (can’t search for the “meaning” of the data)
No resources (time, people, $$$) to build a scalable search service
How to take advantage of the current rise of Generative AI technology

Solution

Scalable, fully-managed out-of-the-box search service
Bring your own web pages, tables, or documents
Provides semantic search capabilities (search by meaning and keywords)
LLM-based summarization, multimodal search, and chatbot integration

Objectives

This blog post will teach you how to perform the following tasks:

Use your own set of unstructured data (PDFs) to build up a knowledge hub
Create your own Search Application with Vertex AI Search (aka Gen AI App Builder)
Test the generated Search Engine
Use the Search Application to summarize your data or ask follow-up questions about the content of your documents

Enough of the introduction, let’s come to the more interesting part, the actual implementation.

Let’s Build

Our Scenario

Throughout this blog post, we’re going to use a very specific dataset from Cymbal, Google’s fictitious company. The dataset contains multiple PDFs about the onboarding process for new employees.

Thus, our goal is to use those PDFs in our datastore and ask our application-specific questions new hires would have about the company. Once we set up that application, we can not only ask about specific information about this dataset but also about the meaning of the content itself.

Enough introduction, let’s start with the interesting part of this post.

Step 1 // Create a new project

Note: If you're already using an existing project, you can skip this step and move on with Step 2.

Create a new project, enable billing, and run the following on your local terminal (or Cloud Shell):

gcloud init

Step 2 // Get your data ready

Note: If you already have a GCS bucket including your PDFs, you can skip this step and continue with Step 3.

Before we can create a search app for asking questions about your data, we need to create a storage in which this data resides. Thus, we take Google Cloud Storage, as it is the most effective option for storing objects within your Google Cloud premises.

If you first want to test this with publicly available sample data before you use your own data, here is a list of multiple datasets from Google that you can use. The last entry is the dataset we’re going to use for our scenario throughout this post:

Step 2.1 // Create a Google Cloud storage bucket

Within the GCP console, use the navigation menu, navigate down to the Storage section, and select Cloud Storage
Create a new bucket for your data by clicking Create
Give your bucket a unique name
Select your bucket configuration as you prefer (depending on your requirements, otherwise just feel free to take the default values)
Click Create to create your GCS bucket

Step 2.2 // Upload your own data (via GCP console)

In the Cloud Storage menu, click Buckets
Select the previously created bucket by clicking on the name
Now, you can upload your data by hitting the Upload Files (or Upload Folder) button
Select the files/folders you want to upload and wait until the upload has been completed

Step 3 // Create a Search App and Data Store

Note: Before you start with this step, ensure you have a dataset ready (either a sample one or your own).

Step 3.1 // Create your Search App

Within the GCP console, use the navigation menu, and navigate down to the Artificial Intelligence section → select Search & Conversation
Activate the needed API by clicking Continue and Activate the API
After the API is activated, the workflow to create a new app will start automatically
(if not, hit New App, this should trigger the creation process manually)
Select Search as the type
Now, you should see two features already pre-selected for you. For this use-case we leave both enabled:

6. Give your app a name and select a location

7. Click Continue

Step 3.2 // Create a Data Store to populate your data

As you have the basics for your new Search app almost ready, now you have to set up the data store in which the data is being populated. Therefore, click Create New Data Store
As Source, select Cloud Storage
Click on Browse right next to the input field of your GCS bucket. This will give you a list of all available storage buckets
Select the storage bucket on which your data resides and click Select.
(If you don’t have your own data or if you just want to use sample data first, you will find a list in Step 2)
Leave the option for the data type on Unstructured documents
Hit Continue and give it a name. It’s generally recommended to use the source type as a prefix, followed by what data is being stored (e.g. gcs-newhire-pdfs)
Hit Create to initiate the creation of the datastore
After that, you will see again the first workflow of creating your app, but with a tiny difference. Now, select the just created datastore and again hit Create

Step 4 // Configure your Search Engine

After the successful configuration of your new datastore, it will take a short while to generate it and set it up. Meanwhile, you can use this time to configure the Search Engine

In the Gen App Builder menu, click Configurations
In the Widget tab, explore the following options for the search type

3. Select Search with follow-ups

4. Ensure that Enable autocomplete stays disabled (not needed in this use-case, as this feature would take about a day to start generating suggestions)

5. Click Save and Publish to save the changes

Step 5 // Verify your new Data Store

In the Vertex AI Search menu, click Data
Check if the data import process is already completed. For this, you can have a look at the following fields:

If the indexing process is still in progress, select the Activity tab

Once the import process is complete, continue with the next step
(if you have chosen one of the sample datasets, then you can grab a coffee meanwhile, as this will take about 10min)

The following screenshot shows a completed, and ready to use, Data Store:

Indicators for a completed upload in Datastore

Step 6 // Test your new Search Engine

Note: As soon as your Data Store has built up his index with the uploaded documents, you can continue with testing your new Search Engine.

In the Vertex AI Search menu, click Preview
Now you should see an input field, asking you to type in your first question about the content
Ask questions about the documents you uploaded before. Here are some suggestions:

I’m a new hire at Cymbal. What do I need to do on my first day?
How would you describe the company curlture?
How do I book my business flights once I have to visit customers?
Can I work from home or do I need to be in the office all the time?
…

Asking our first questions in Vertex AI Search

At this point, feel free to play around and use your creativity for that task. If you have set up everything as mentioned before, you should now also be able to ask follow-up questions to the answers you receive

asking follow-up questions in Vertex AI Search

Now that you set up and tested your new application, you might also be interested in going a step further and actually integrating it into a web application. The following bonus material section will guide you through that part.

Intermediate Tech skills are required from this point onwards

Bonus Material

Step 8 // Integrate the Search App in your web applications

In case you also want to implement that powerful logic into your already existing web application, then this bonus section might also be of interest to you.

In our scenario, we’re building upon a sample web app provided by Google’s developer advocates. Feel free to adapt it to your own web app.

This repository also includes many other examples in regards to Generative AI. So in case you want to know more about the other GenAI options from Google, this is a perfect place to start and try implementing.

However, in this blog, we’ll be focusing on only the search capabilities of GenAI. Other features might be covered in our upcoming blog posts.

Technologies used

Vertex AI Search
Cloud Run

Requirements

What do we need before starting:

Install Python
Install Google’s Cloud SDK

Setup

Clone the repo onto your machine (or use Cloud Shell)

git clone https://github.com/GoogleCloudPlatform/generative-ai.git

2. As this repo also contains many other examples about GenAI, switch to the according sample web app once you cloned the repository

cd generative-ai/search/web-app

3. The structure should look something like the following

folder structure of cloned sample web app

4. Install the needed dependencies

pip install -r requirements.txt

5. Enable the API

gcloud services enable discoveryengine.googleapis.com
gcloud services enable run.googleapis.com

6. Setup application default authentication

gcloud auth application-default login

Deployment

Setup Vertex AI Search Integration

Switch to the GCP console
In the Vertex AI Search menu, click Apps, select your previously created app, and click Integration in the left menu
Set the authorization to Public Access. This will allow you to test our search app before setting any further restrictions. Don't worry about the allowed domains in the next section yet. This will be provided, once we have our Cloud Run instance (and therefore a valid URL to allowlist) ready. Leave it empty for now, and hit Save before you continue

select authorization type to public access

4. As we’re going to use the prebuilt widget, provided in the GCP console, copy the configId from the <gen-search-widget>section

5. Watch out, that you only copy the value of the configIdfield

6. In consts.pyupdate your configuration with your PROJECT_IDand LOCATION. Also, you need to add the configIdto the WIDGET_CONFIGSobject, alongside some other parameters, such as a name and the placeholder. Below you see an example of what it can look like.
Note: Watch out as the number you copied before calls the ID configID, whereas in this python script it's called config_id.

7. There are a bunch of more examples in that consts.py file. If you want to add any more Search apps, the included placeholder values will show you how it is done. If not, you can leave the rest as is

Deploy your Cloud Run App

Now it’s time to deploy the application. Thus, we will use all of the code we have and create a Cloud Run instance of it

gcloud run deploy vertex-ai-search-demo --source .

2. If you have to select a region, then select one of your choice by entering the according number and hit return

3. Next, allow unauthenticated invocations to this instance by entering “y”, as we want to make it publicly available for testing purposes

4. This process shouldn’t take too long (~ 5min)

5. After you have successfully deployed your Cloud Run instance, you should see the URL to access your newly deployed instance

building & deploying your new search app as a Cloud Run instance

6. Use the generated Service URL and add it to the allowed domains list (GCP Console > Search & Conversation > Select your created search app > Integration > Add Service URL to allowed domains list > Save)

7. Now browse the Service URL with your Browser to visit the deployed instance

There you go, here is your sample web application ready to use and explore

Conclusion

In summary, Vertex AI Search significantly streamlines the way we interact with and extract value from data within documents. This powerful yet intuitive platform empowers developers to rapidly craft and deploy semantic search applications, leveraging the forefront of Google’s AI advancements. With the convenience of pre-trained Generative AI models and the flexibility to incorporate both structured and unstructured data, Vertex AI Search effortlessly minimizes the complexity traditionally involved in customizing a search experience.

Throughout the main part of this post, we’ve explored a detailed walkthrough, set in an illustrative scenario, that guides you through the process of establishing a new-hire search tool. This tool is designed to effortlessly answer queries from new employees about their initial days at work. We’ve outlined the essential steps to harness this innovative technology: from initiating a search application to enriching it with relevant data.

Additionally, we provided insights into how this search functionality can be seamlessly integrated into existing web applications, ensuring a cohesive and efficient user experience. This integration not only enhances the usability of web platforms but also maximizes the utility of the search tool itself, making it a vital asset for any organization looking to improve its data accessibility and knowledge management.

Future Work

You might ask what else you could build with applications like this one. Well, search applications like the one covered here, also open the door for a wide range of other search applications, such as:

Product search engines: Build your own product search engines that can recommend products to customers based on their individual needs and preferences
Knowledge Base: Build a knowledge base that can help people learn new things and find answers to their question

This article is part of our effort and commitment to finding unique cloud solutions to enhance our customer’s experience.

Stay tuned for more insights! Follow and subscribe to happtiq!

Author: Alex Hirz