Journey to Personal AI: Enhancing Personal Knowledge Management with Khoj
The Journey Towards Personal AI: From Screening Agents to Digital Twins and AI-Powered Knowledge Bases
Personal AI is a term that is used in many ways. The concept generally refers to artificial intelligence systems designed to interact with individuals on a personal level, learning and adapting to their specific needs and preferences. This could be in the form of a digital assistant that screens calls, emails, and texts, or a more advanced system that can respond to and complete transactions, answer queries, and even extend a person’s digital presence.
The development of personal AI can be seen as a journey unfolding in several stages. The first stage involves the AI acting as a screening agent, filtering through digital communications such as calls, emails, and texts. This ensures that only relevant and important information requires the user’s attention, saving valuable time and reducing digital clutter.
As the AI evolves, it enters the second stage, where it can respond to and complete simple transactions, such as scheduling a car service. The third stage sees the AI tackling knowledge transactions, like answering queries and providing information. The final stage extends the user’s presence, with the AI stepping in to assist in situations where the user cannot physically be present or is otherwise occupied. This could include acting as a digital twin, replicating the user’s responses and decisions.
Conversational AI, a subfield of artificial intelligence, is a key component of personal AI. It focuses on producing natural and seamless conversations between humans and machines, simulating how a human would behave in a conversation and providing appropriate responses to the user’s queries or statements. These systems can understand and respond to text or voice inputs from users and can be used across multiple industries for various tasks like customer service, advice, or general inquiries.
However, it’s important to remember that while personal AI can be incredibly useful, it should be used as a complement to human intelligence and decision-making, not as a replacement.
One of the baby steps towards personal AI is a personal knowledge base powered by AI. This system uses artificial intelligence to organize, manage, and retrieve information that is personally relevant to an individual. This could include anything from personal notes and documents to more complex forms of data like health metrics or financial information.
The AI in these systems can help categorize and tag information, making it easier to find when needed. It can also identify patterns and relationships in the data, providing insights that can help the user make better decisions or improve their understanding of a particular topic.
In the context of personal AI, an AI-powered knowledge base can serve as a valuable tool for managing and making sense of the vast amounts of digital information that individuals deal with daily. It can also act as a foundation for more advanced forms of personal AI, providing the data and insights that these systems need to learn and adapt to the user’s needs.
There are several tools and platforms available that claim to provide a comprehensive solution for personal AI or at least for a personal knowledge base. In a series of upcoming articles, I will cover these tools and platforms.
About Khoj
Building on the concept of personal AI, Khoj offers a decent application designed to enhance personal knowledge management.
Khoj is an open-source AI copilot designed to augment personal knowledge management systems. Key features include:
- AI Copilot: Provides contextual AI assistance across different platforms.
- Second Brain Integration: Enhances personal note-taking and knowledge management systems.
- Open-Source: Allows customization and transparency.
- Cross-Platform Support: Works on various devices and operating systems.
- Contextual Understanding: Understands the context of user queries to provide relevant assistance.
Running Khoj with hosted API and locally with LM Studio
In this article, we will explore how to install Khoj and work entirely locally using LM Studio.
Installation
Khoj provides two options for self-hosting on local machines:
— Utilizing Docker
— Operating in a Python and PostgreSQL environment.
Between these two options, I found the second one to be more efficient (on a Mac air), so I will detail those steps below.
- Download and install postgressapp from https://postgresapp.com/ that comes pre-installed with
pgvectorand relevant dependencies.
2. Double click on postgres to launch a terminal and create a database ‘khoj’ using this command ‘CREATE DATABASE khoj;’.
3. Create a separate python environment (I use anaconda navigator) and install khoj with the following.
# ARM/M1+ Machines
MAKE_ARGS="-DLLAMA_METAL=on" python -m pip install khoj-assistant
# Intel Machines
python -m pip install khoj-assistantThese three steps are pretty much it, from installation standpoint you could refer to the below link if any issues.
Once properly installed, the following steps will run the Khoj server in the background.
- Configure the environment variables
export KHOJ_ADMIN_EMAIL=<your-email>
export KHOJ_ADMIN_PASSWORD=<your-password>2. Run the server
khoj --anonymous-mode--anonymous-mode allows you to run the server without setting up Google credentials for login.
3. Khoj should now be running at http://localhost:42110. You can see the web UI in your browser.
Note: localhost as opposed to 127:0:0:1, or 0:0:0:0 is more reliable, even though the server displays the same in the terminal.
Setup
- Login to admin panel http://localhost:42110/server/admin (using the previous credentials
- Create a ‘ChatModelOptions object’ as under.
Chat model is the local model you will connect to in LM Studio
Model Type is Openai, (You will use the same for Ollama)
3. Map the above to a ‘OpenAIProcessorConversationConfig object’
Api key and Api base url are the default values as prescribed in LM Studio
4. Assign the Chat model object to the agent.
Khoj comes with a default agent.
Chat model is the ChatModelOptions object you would like to assign.
Note: One can set up multiple agents and attach different models. If you are unsure about Tools, just use the default value [“*”]
Execution-With LM Studio
In this test, I chose two AI models to suit my measly configuration of MacAir :-)
Using the Local model as knowledge base.
- lmstudio-ai/gemma-2b-it-GGUF/gemma-2b-it-q8_0.gguf
- MaziyarPanahi/Meta-Llama-3–8B-Instruct-GGUF/Meta-Llama-3–8B-Instruct.Q3_K_S.gguf
- The web interface of Khoj offers several routing options for the prompts.
For local models, the best way to get started is /general.
Here is a sample prompt ‘give me a brief paragraph on Albert Einstein’ and output for the two models.
Using the local files as knowledge base.
- Use the same web interface to upload the file for indexing.
2. Once done, you can inspect the results in the admin panel under ‘Entrys’
Note: For some reason, I was unable to get a response from my personal knowledge base using /notes, even though the indexed file segments and related embeddings were displayed.
Execution-With openAI API
I was able to get it working with openAI API key, following the same steps for creating an agent>ChatModelOptions object>OpenAIProcessorConversationConfig object, using the same uploaded file/s.
Here are two sample outputs.
[04:18:08.002534] INFO khoj.routers.api: 🔍 Searching knowledge base with queries: ['Quotes by Swami Sivananda to Westerners'] api.py:362
[04:18:30.201994] INFO khoj.processor.conversation.utils: First response took: 0.967 seconds utils.py:59
[04:18:30.469300] INFO khoj.processor.conversation.utils: Chat streaming took: 1.234 seconds utils.py:48
[04:18:30.486027] INFO khoj.processor.conversation.utils: Saved Conversation Turn utils.py:128
You (default): "What did Swami Sivananda say to Westerners?"
Khoj: "Swami Sivananda encouraged Westerners to serve, love, give, purify, meditate, and realize. He emphasized these principles as a way of life." 
[04:30:58.770091] INFO khoj.routers.api: 🔍 Searching knowledge base with queries: ['Teachings of Karma Yoga on detaching from outcomes'] api.py:362
[04:31:02.842612] INFO khoj.processor.conversation.utils: First response took: 0.914 seconds utils.py:59
[04:31:04.255465] INFO khoj.processor.conversation.utils: Chat streaming took: 2.328 seconds utils.py:48
[04:31:04.267819] INFO khoj.processor.conversation.utils: Saved Conversation Turn utils.py:128
You (default): "What is the best way to detach oneself from outcomes, as per Karma Yoga?"
Khoj: "In Karma Yoga, the best way to detach oneself from outcomes is to focus on performing actions selflessly, without attachment to the results or fruits of
those actions. By dedicating your actions to a higher purpose or to the service of others, and by maintaining a mindset of offering your efforts without expecting
personal gain, you can gradually cultivate a sense of detachment from the outcomes. This practice helps in purifying the mind and reducing the ego's influence,
leading to inner peace and spiritual growth.
Would you like to explore specific techniques or practices within Karma Yoga that can help you detach yourself from outcomes further?" Note: I was unable make the /notes feature with OpenRouter API as well.
Github as a datasource
Khoj currently allows Notion and Github as data sources in addition to your personal files.
Setting up Github is very easy.
- Generate a classic PAT (personal access token) from Github with
repoandadmin:orgscopes at least. - Navigate to https://app.khoj.dev/config/content-source/github to configure your Github settings. Enter in your PAT, along with details for each repository you want to index.
- Click
Save. - Go to https://app.khoj.dev/ and start chatting!
You can cross check the ‘GithubConfig object’ in the admin panel.
….and the entries in the db.
Once again I used openAI model(gpt-3.5-turbo) to chat with this data.
If you prefer a simpler way of accessing your data and aren’t concerned about where it is stored, you can use Khoj Desktop and Khoj Cloud, which offer a free tier to get started. Currently, Obsidian plugin access is only available for Khoj Cloud users (including those using the free tier).
In self-host scenario, if you are interested in basic chatting or searching within Obsidian docs, one workaround is to sync your vault with Github and use that as a data source.
In conclusion, Khoj offers a comprehensive suite of features for personal knowledge management and AI-powered information retrieval. Currently, the cloud variant seems more beneficial due to its capabilities, integrations and user-friendly interface. Hope to see the ‘completely-local’ variant, with its local models, to become increasingly feasible and attractive in the coming months. This would give users complete control over their data, enhancing privacy and security and developers to create more functionalities.
