SteamDeck and a GPT model — on the brink of absurdity
I’ve been late to the recent party of AI and GPTs. When I tried ChatGPT it seemed impressive and promising, but a sobering thought buried my hopes and dreams for the bright future of AI. I realized it was unusable for my software engineering job. Although the abilities of GPT’s could definitely be useful in programming, the legal side of such cooperation is a big problem. All of that turned me away from the AI at that moment.
Some time later I discovered a cure for my lack of AI enthusiasm — GPT4ALL by NomicAI. Based on the llama-7B, this AI model is tuned to run on consumer-grade CPUs. Yes, it is not ChatGPT, not anywhere close, but you’ll be impressed what a large language model 4GB of size and your office laptop are capable of without any connection to the internet! Not to mention you keep all the data on your machine.
So, I’ve been playing with GPT4ALL on my M1 pro MacBook for a while now, and the performance has been really good. The other thing I’ve been playing with recently is my 64GB SteamDeck. So, why not fuse the fun and run the AI model on my gaming console?
Setup
Step 1
Basically you’ll need two things: the model itself and software to communicate with the model.
Download the software: https://github.com/nomic-ai/gpt4all/raw/main/chat/gpt4all-lora-quantized-linux-x86
Download the model: direct link or torrent
You can do it ether right on a SteamDeck or on a computer and then transfer the files to your SteamDeck.
Step 2
Switch SteamDeck to the desktop mode and place the files in the same directory then open a terminal in that directory.
Run the following command to start the whole thing:
./gpt4all-lora-quantized-linux-x86 -m gpt4all-lora-quantized.bin -t 4‘-m’ specifies a path to the model
‘-t’ specifies the number of cpu cores that will be utilized to run the inference
Feel free to play around with other params like ‘temp’ to adjust creativeness/precision of the model or other params. To get a full list of params, run the following command:
./gpt4all-lora-quantized-linux-x86 -m gpt4all-lora-quantized.bin -hStep 3
Run your prompts and have fun! Playing with AI is no less fun than playing games, I assure that.
Conclusion
GPT4ALL is already impressive — your own AI chat bot that you can run without a million dollar GPU cluster, without internet connection, without disclosing any information to any third party. On top all of that, the fact that even a SteamDeck can run the model is just stunning.
Just stop and think for a moment, what seemed like magic a couple years ago is now accessible to everyone and you are free to perform these miracles on your handheld gaming console!
