Using Pinecone with Bedrock Knowledge Base

As the Gen-AI era has emerged, vector databases have gained prominence as well. Hallucination — providing wrong information that appeared to be plausible — is a commonly faced issue when it comes to generative AI applications.

By storing relevant data in a vector database and having a large language model (LLM) generate answers based on this data, users can receive more accurate responses based on their data.

Pinecone is a managed vector database on the cloud. You can easily save and query vector data in Pinecone. If you have data pipeline deployed in AWS and simply want to expand this pipeline to Generative AI, you can simply connect to Pinecone from AWS data services such as SageMaker, Glue notebook etc.

Here I will show you how you can use Pinecone for RAG and how Bedrock Knowledge base can simplify the same process.

Use Pinecone with Jupyter Notebook

I created a SageMaker notebook instance and made sure that the IAM role has permission to access S3. If SageMaker is located in private subnet, private endpoint is required. I launched the instance in public subnet as it’s just for simple test. When the proper permission is granted to the IAM role that is attached to SageMaker, we are good to go.

!pip3 install awswrangler
import awswrangler as wr

df = wr.s3.read_csv("s3://test/pinecone/articles1.csv")

I used AWS wrangler to read data from S3.

Here is a sneak peek at the data that I downloaded from Kaggle. The data contains diverse articles covering different domains. If you want to insert data into Pinecone, the ‘id’ and ‘values’ columns are required.

#import necessary libraries
!pip3 install sentence_transformers

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2', device = 'cpu' )

#prepare values data to insert
df['content'] = df['content'].map(lambda x: (model.encode(x)).tolist())
df = df.rename(columns={'content':'values'})

The data to insert into vector database should be in vector format. After I chose a model for vector embedding, I converted the ‘content’ column into a vector and renamed it to ‘values’ using the embedding model.

#prepare metadata to insert
df['metadata'] = df['content'].str[:500]
df['metadata'] = df.apply(lambda x: {
    'content': x['metadata'],
}, axis = 1)

#prepare id to insert
df['id'] = df['Unnamed: 0']
df['id'] = df['id'].map(lambda x:str(x))

#prepare final data to insert
df_upsert = df[['id', 'values','metadata']]

Besides the necessary data, I wanted to add metadata so that I can have better view for the query response. As metadata has to be in dictionary format, I added a ‘metadata’ column in dictionary format.

Now the ‘id’, ‘values’ and ‘metadata’ columns are ready. This is how my data looks like after all the processing.

pc = Pinecone(api_key = "${api key}")
pc.create_index(
                name = "news-index", 
                dimension = 384,
                metric = "cosine",
                spec = PodSpec(
                    environment = "gcp-starter"
                )
               )
idx = pc.Index('news-index')

I created a Pinecone index and the object. An index in Pinecone is a collection of vectors and similar to a table in RDB. I created a Pinecone index and specified a metric. There are three metrics provided from Pinecone — Euclidean, cosine and dot products. As I’ll query vector data to retrieve relevant data, I chose cosine as the metric.

I could check the newly created index from the Pinecone console.

idx.upsert_from_dataframe(df_upsert)

Now it’s time to upsert data into Pinecone index.

The data is all upserted into the Pinecone index.

res = idx.query(vector = (model.encode("Tell me about USA president in 2017.")).tolist(), ##it has to be list 
               top_k=10, include_metadata=True)

We’re almost there! I couldn’t wait to run query. The query also needs to be converted into a vector.

And here I got the result, which is relevant article title to my question.

So this is how you do vector embedding, upsert the vectors into a vector database, and finally query the vectors from the database.

Use Pinecone with Bedrock Knowledge Base

You could see that vector embedding and query isn’t as difficult as it sounds. BUT, with Bedrock Knowledge Base, it makes it even easier. Now let’s see how we can use Pinecone with Bedrock Knowledge Base.

Before we start creating Knowledge Base, we need to create a new secret in Secrets Manager so that it has the API key to access Pinecone safely.

Choose Other type for Secret type and set ‘apiKey’ as the key. Fill your Pinecone api key in the value box. After giving a name, leave everything as default and create the Secret.

This Secret ARN is required when creating Bedrock Knowledge Base.

And now it’s time to create the Knowledge Base. First move to Knowledge Base tab in Amazon Bedrock page and create a Knowledge Base.

I gave a name for the Knowledge Base and granted permissions. It requires three main permissions — read from S3, invoke the Bedrock model and get the value from Secrets Manager. Alternatively, you can simply create a new role through the UI.

I specified the S3 path where the data to be inserted into vector database is located in S3 URL field.

In this step, I chose a model and vector database to use. I filled in Pinecone index host in Endpoint URL box and Secrets ARN in Credentials secret ARN box that I got earlier.

The Text field requires field name which needs to be converted in to a vector and queried.

After reviewing all the configuration, I created Knowledge Base.

When the Knowledge base is created, click the Sync button to write data from S3 to the vector database.

Voilà! After syncing is completed, I asked a simple question, and it responded based on the data I provided. My question was ‘What happened in USA in 2017?’ and it responded, ‘Donald Trump was inaugurated as the 45th President of the United States on January 20, 2017.’

With Knowledge base, it handles converting and upserting vectors into the database. It makes the whole process ofconfiguring vector database much easier.

Errors You May Encounter

The knowledge base storage configuration provided is invalid… The vector database encountered an error while processing the request.

When I tried to create the Knowledge Base, it kept showing me the error above. Turns out it was dimensions mismatch. My Pinecone vector index had 384 dimensions as it used all-MiniLM-L6-v2 embedding model and Titan Embeddings G1 creates 1536 dimensions. After recreating the index with proper dimensions, Knowledge Base was successfully created.

AI applications are powerful yet versatile tools that can significantly improve work efficiency. As many experts advise, we should embrace and learn to effectively utilize these technologies. In line with this trend, a growing number of tools like Bedrock are becoming available, enabling us to harness the potential of AI and integrate it seamlessly into our workflows.