Quick tip: How to Build Local LLM Apps with Ollama and SingleStore

Use Ollama with SingleStore

Abstract

In an era of heightened data privacy concerns, the development of local Large Language Model (LLM) applications provides an alternative to cloud-based solutions. Ollama offers one solution, enabling LLMs to be downloaded and used locally. In this short article, we’ll explore how to use Ollama with LangChain and SingleStore using a Jupyter Notebook.

The notebook file used in this article is available on GitHub.

Introduction

We’ll use a Virtual Machine running Ubuntu 22.04.2 as our test environment. An alternative would be to use venv.

Create a SingleStoreDB Cloud account

A previous article showed the steps required to create a free SingleStore Cloud account. We’ll use Ollama Demo Group as our Workspace Group Name and ollama-demo as our Workspace Name. We’ll make a note of our password and host name. For this article, we’ll temporarily allow access from anywhere by configuring the firewall under Ollama Demo Group > Firewall. For production environments, firewall rules should be added to provide increased security.

Create a Database

In our SingleStore Cloud account, let’s use the SQL Editor to create a new database. Call this ollama_demo, as follows:

CREATE DATABASE IF NOT EXISTS ollama_demo;

Install Jupyter

From the command line, we’ll install the classic Jupyter Notebook, as follows:

pip install notebook

Install Ollama

We’ll install Ollama, as follows:

curl -fsSL https://ollama.com/install.sh | sh

Environment Variable

Using the password and host information we saved earlier, we’ll create an environment variable to point to our SingleStore instance, as follows:

export SINGLESTOREDB_URL="admin:<password>@<host>:3306/ollama_demo"

Replace <password> and <host> with the values for your environment.

Launch Jupyter

We are now ready to work with Ollama and we’ll launch Jupyter:

jupyter notebook

Fill out the notebook

First, some packages:

!pip install langchain ollama --quiet --no-warn-script-location

Next, we’ll import some libraries:

import ollama
from langchain_community.vectorstores import SingleStoreDB
from langchain_community.vectorstores.utils import DistanceStrategy
from langchain_core.documents import Document
from langchain_community.embeddings import OllamaEmbeddings

We’ll create embeddings using all-minilm (45 MB at the time of writing):

ollama.pull("all-minilm")

Example output:

{'status': 'success'}

For our LLM we’ll use llama2 (3.8 GB at the time of writing):

ollama.pull("llama2")

Example output:

{'status': 'success'}

Next, we’ll use the example text from the Ollama website:

documents = [
    "Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels",
    "Llamas were first domesticated and used as pack animals 4,000 to 5,000 years ago in the Peruvian highlands",
    "Llamas can grow as much as 6 feet tall though the average llama between 5 feet 6 inches and 5 feet 9 inches tall",
    "Llamas weigh between 280 and 450 pounds and can carry 25 to 30 percent of their body weight",
    "Llamas are vegetarians and have very efficient digestive systems",
    "Llamas live to be about 20 years old, though some only live for 15 years and others live to be 30 years old"
]

embeddings = OllamaEmbeddings(
    model = "all-minilm",
)

dimensions = len(embeddings.embed_query(documents[0]))

docs = [Document(text) for text in documents]

We’ll specify all-minilm for the embeddings, determine the number of dimensions returned for the first document, and convert the documents to the format required by SingleStore.

Next, we’ll use LangChain:

docsearch = SingleStoreDB.from_documents(
    docs,
    embeddings,
    table_name = "langchain_docs",
    distance_strategy = DistanceStrategy.EUCLIDEAN_DISTANCE,
    use_vector_index = True,
    vector_size = dimensions
)

In addition to the documents and embeddings, we’ll provide the name of the table we want to use for storage, the distance strategy, that we want to use a vector index and the vector size using the dimensions we previously determined. These and other options are explained in further detail in the LangChain documentation.

Using the SQL Editor in SingleStore Cloud, let’s check the structure of the table created by LangChain:

USE ollama_demo;

DESCRIBE langchain_docs;

Example output:

+----------+------------------+------+------+---------+----------------+
| Field    | Type             | Null | Key  | Default | Extra          |
+----------+------------------+------+------+---------+----------------+
| id       | bigint(20)       | NO   | PRI  | NULL    | auto_increment |
| content  | longtext         | YES  |      | NULL    |                |
| vector   | vector(384, F32) | NO   | MUL  | NULL    |                |
| metadata | JSON             | YES  |      | NULL    |                |
+----------+------------------+------+------+---------+----------------+

We can see that a vector column with 384 dimensions was created for storing the embeddings.

Let’s also quickly check the stored data:

USE ollama_demo;

SELECT SUBSTRING(content, 1, 30) AS content, SUBSTRING(vector, 1, 30) AS vector FROM langchain_docs;

Example output:

+--------------------------------+--------------------------------+
| content                        | vector                         |
+--------------------------------+--------------------------------+
| Llamas weigh between 280 and 4 | [0.235754818,0.242168128,-0.26 |
| Llamas were first domesticated | [0.153105229,0.219774529,-0.20 |
| Llamas are vegetarians and hav | [0.285528302,0.10461951,-0.313 |
| Llamas are members of the came | [-0.0174482632,0.173883006,-0. |
| Llamas can grow as much as 6 f | [-0.0232818555,0.122274697,-0. |
| Llamas live to be about 20 yea | [0.0260244086,0.212311044,0.03 |
+--------------------------------+--------------------------------+

Finally, let’s check the vector index:

USE ollama_demo;

SHOW INDEX FROM langchain_docs;

Example output:

+----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+
| Table          | Non_unique | Key_name   | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type       | Comment | Index_comment | Index_options                         |
+----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+
| langchain_docs |          0 | PRIMARY    |            1 | id          | NULL      |        NULL |     NULL |   NULL |      | COLUMNSTORE HASH |         |               |                                       |
| langchain_docs |          1 | vector     |            1 | vector      | NULL      |        NULL |     NULL |   NULL |      | VECTOR           |         |               | {"metric_type": "EUCLIDEAN_DISTANCE"} |
| langchain_docs |          1 | __SHARDKEY |            1 | id          | NULL      |        NULL |     NULL |   NULL |      | METADATA_ONLY    |         |               |                                       |
+----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+

We’ll now ask a question, as follows:

prompt = "What animals are llamas related to?"
docs = docsearch.similarity_search(prompt)
data = docs[0].page_content
print(data)

Example output:

Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels

Next, we’ll use the LLM, as follows:

output = ollama.generate(
    model = "llama2",
    prompt = f"Using this data: {data}. Respond to this prompt: {prompt}"
)

print(output["response"])

Example output:

Llamas are members of the camelid family, which means they are closely related to other animals such as:

1. Vicuñas: Vicuñas are small, wild relatives of llamas and alpacas. They are native to South America and are known for their soft, woolly coats.
2. Camels: Camels are also members of the camelid family and are known for their distinctive humps on their backs. There are two species of camel: the dromedary and the Bactrian.
3. Alpacas: Alpacas are domesticated animals that are closely related to llamas and vicuñas. They are native to South America and are known for their soft, luxurious fur.

So, in summary, llamas are related to vicuñas, camels, and alpacas.

Summary

In this short article, we’ve seen that we can connect to SingleStore, store the documents and embeddings, ask questions about the data in the database, and use the power of LLMs locally through Ollama.