Training an LLMs model Achieving 99.3% of ChatGPT’s Performance on Consumer-Level Hardware

Large Language Models (LLMs) are powerful tools in the field of artificial intelligence. They can generate human-like text, answer questions, translate languages, and much more. However, these models are often very large, making them difficult to run or train on standard consumer hardware. This has been a significant challenge for users and accessibility.

Hugging Face, a leading AI company, has been working on making these models more accessible. They have collaborated with bitsandbytes to allow users to run models in 4-bit precision. This includes a large majority of Hugging Face models, in any modality (text, vision, multi-modal, etc.). Users can also train adapters on top of 4bit models leveraging tools from the Hugging Face ecosystem. This is a new method introduced in the QLoRA paper by Dettmers et al.

What is QLoRA?

QLoRA stands for Quantized Low-Rank Adapters. It is an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.

QLoRA uses 4-bit quantization to compress a pre-trained language model. The language model parameters are then frozen, and a relatively small number of trainable…