How to Augment LLMs with Private Data

On why you needs this

Language Models have become powerful stores of general knowledge and the latest models can answer questions across a wide range of topics. Many are calling it a revolution and some going as far as saying that if you don’t adopt it, you and your team will fall to the wayside.

But whether you want to use LLMs to help your customers or to improve internal team efficiency, you’ll find that usually, the first step is to provide the LLM with more context. This might be your product website, internal communication channels, wikis/documentation, code base, or a number of other sources.

This way, you can go from:

LLMs aren’t mind readers

To something useful like:

But a system connected to your knowledge can answer this and much more

Sounds great, how do I get there?

Option 1: Build it Yourself

Finally it’s possible, but it still takes expertise in NLP, information retrieval, and system architecture. With amazing tools like Llama Index and LangChain, it could take as little as 2 days to build a prototype, but it often still takes months to get something production ready.

Option 2: Adopt a Prebuilt Solution

With the wave of excitement stemming from ChatGPT, there are a number of startups that are building services based on LLMs ranging from customer support bots to universal search. This takes the burden of building and maintenance off of your team but if you need specific features or the company building the service goes in a different direction, there’s not much you can do.

Option 3 (this one’s a winner): Adopt an Opensource Solution

Without any upfront investment, you get a product that is production ready and flexible enough to extend in whichever way your team needs. These projects are also generally fast to pick up the latest technology and methodologies thanks to outside contributors. Just be sure that the project has an active core team that is responsive to its users.

Connecting to your sources

This guide will go over connecting an opensource project called Danswer to your private tools.

But before we get started, why is Danswer a good choice for this?

• It is fully opensource (MIT License) and free to use.
• Allows you to plug and play different LLM models like GPT, HuggingFace, GPT4All, Llama cpp, or even custom self hosted models.
• Has crucial features out of the box like document access control, frontend UI, management dashboards, polling for document updates, and flexible deployment options.
• A decent list of connectors to other tools like Slack, GitHub, GoogleDrive etc.

By default settings Danswer uses the GPT line of models from OpenAI but in the spirit of opensource, this guide will show how to set up Danswer up with an opensource model, Llama 2.

Setting up a Llama 2 Endpoint

If you haven’t seen my previous post on deploying Llama 2, I recommend starting there as it explains everything from this section in more detail.

A notebook is provided below to deploy Llama 2 on a Nvidia T4 GPU for free via Google Colab.

HostLlama2API

Note: Colab GPU instances will be reclaimed if the user is not actively using the colab notebook.

Once you reach the cell shown in the following image, note the public URL of your Llama 2 model server.

Note: this way of hosting a LLM is mostly for demo/development purposes as this is the only way to do it completely for free. However Danswer offers trivial ways to connect to Azure APIs, OpenAI APIs, HuggingFace Endpoints/APIs, with GCP and AWS coming soon.

Deploying Danswer

Danswer provides Docker containers which makes it easy to deploy on any cloud whether on a single instance or via Kubernetes. For this demo, we will use Docker Compose to get Danswer running locally.

First pull the code:

git clone https://github.com/danswer-ai/danswer.git

Next navigate to the deployment directory (shown for Linux):

cd danswer/deployment/docker_compose

Configure Danswer to use the new Llama 2 endpoint by creating an .env file to overwrite some defaults (shown for Linux):

GEN_AI_MODEL_PROVIDER=custom  # Sets Danswer to use custom requests rather than a client library to interface with LLM
GEN_AI_API_ENDPOINT=<REPLACE-WITH-YOUR-NGROK-PUBLIC-URL>/generate  # Sets the model endpoint URL

Start up Danswer (uses docker version ≥= 1.13.0):

docker compose -f docker-compose.dev.yml -p danswer-stack up -d --pull always --force-recreate

Once this finishes, Danswer will be available at http://localhost:3000

More docs on Danswer are available https://docs.danswer.dev/quickstart

Adding Documents to Danswer

Connecting different tools to Danswer is done via the admin page.

The Admin Panel is found on the top right

You can also find docs under https://docs.danswer.dev/connectors/overview for each of the supported connectors.

For this demo, we will index a public website. Since we’ve referred to the Danswer documentation multiple times, let’s go ahead and index that.

Provide the URL and click Connect

Getting Answers from Danswer

Click on the Danswer logo to get back to the home page and now you can ask questions regarding the newly indexed documentation.

Answers are backed by sources and other relevant documents are found under Results

Closing Comments

Another useful resource for Danswer is the project’s Slack workspace and Discord server for the questions/support. The opensource community around the project is also great about helping out new members.

Again, the Danswer code is completely opensource and you can find it here: https://github.com/danswer-ai/danswer

I hope you found this blog informative!