Run PrivateGPT on Windows

Tired of finding ways to run privateGPT on windows? read this.

Honestly, I’ve been patiently anticipating a method to run privateGPT on Windows for several months since its initial launch. Whether it’s the original version or the updated one, most of the tutorials available online focus on running it on Mac or Linux. It was only yesterday that I came across a tutorial specifically for running it on Windows. As a gesture of appreciation, I invite you to check out the tutorial creator’s YouTube channel:

The installation procedures were given in privateGPT Installation Guide for Windows Machine (PC) — simplifyai.in

Firstly, I want to share some information that is seldom discussed in most tutorials — the laptop specifications required to run it. I personally run this on a recently acquired ASUS TUF F15 equipped with a 12th gen Intel Core i7–12700H (20 CPUs) running at approximately 2.3GHz, 32GB of RAM, and an NVIDIA GeForce RTX 4060 with 8GB of VRAM.

Below is the VRAM requirement for different models depending on their size (Billions of parameters). The estimates in the table does not include VRAM used by the Embedding models — which use an additional 2GB-7GB of VRAM depending on the model.

localGPT/ at main · PromtEngineer/localGPT (github.com)

Given that it’s a brand-new device, I anticipate that this article will be suitable for many beginners who are eager to run PrivateGPT on their Windows machines.

Use MiniConda instead of Anaconda

1. Download the Miniconda installer for Windows
2. Run the installer and follow the on-screen instructions to complete the installation. Make sure to check the box that says “Add Miniconda3 to my PATH environment variable” during installation.
3. After installation, close and reopen your terminal to make sure the changes take effect.

Miniconda — miniconda documentation

Make sure to thick this second checkbox

After that, open the anaconda prompt as administrator

Then follow these steps:

Create the environment using python 3.11

conda create -n privateGPT python=3.11

Click proceed

Activate the environment.

conda activate privateGPT

Download the github

imartinez/privateGPT: Interact with your documents using the power of GPT, 100% privately, no data leaks (github.com)

Extract dan simpan direktori penyimpanan

Change directory to said address.

Install packages.

conda install -c conda-forge pipx

Install Poetry

pipx install poetry

set “PATH=%USERPROFILE%\.local\pipx\venvs\poetry\Scripts;%PATH%”

Install Poetry ui

poetry install — with ui,local

wait until it is completed

Run the scripts.

cd scripts
ren setup setup.py
cd ..
poetry run python scripts/setup.py
set PGPT_PROFILES=local
set PYTHONPATH=.
poetry run python -m uvicorn private_gpt.main:app --reload --port 8001

set PGPT and Run

Wait until it’s finished.

Wait for the model to download, and once you spot “Application startup complete,” open your web browser and navigate to 127.0.0.1:8001.

Digesting document

Query Document

Additional Notes:

• Changing the Model: Modify settings.yaml in the root folder to switch between different models.
• Running on GPU: If you want to utilize your GPU, ensure you have PyTorch installed.

GPU Setup Commands

1. Install PyTorch with CUDA support:

pip install torch==2.0.0+cu118 --index-url https://download.pytorch.org/whl/cu118

1. Set CMake arguments for llama-cpp-python:

$env:CMAKE_ARGS='-DLLAMA_CUBLAS=on'; poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python

Run PrivateGPT with GPU Acceleration

Now, launch PrivateGPT with GPU support:

poetry run python -m uvicorn private_gpt.main:app --reload --port 8001

Additional Notes:

• Verify that your GPU is compatible with the specified CUDA version (cu118).
• Ensure that the necessary GPU drivers are installed on your system.