The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

Image by:

AI World Vision

Abstract

Recent research, exemplified by BitNet 1, is ushering in a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a variant called BitNet b1.58, where every single parameter (or weight) of the LLM is ternary: {-1, 0, 1}. BitNet b1.58 achieves performance equivalent to full-precision (FP16 or BF16) Transformer LLMs with…