BitNet b1.58: How Ternary Weights Cut LLM Memory by 82% Without Losing Accuracy

Every large language model carries its weight literally: a 7-billion-parameter model in 16-bit floating point occupies roughly 14 GB of memory before a single token is processed. Scaling to 70 billion parameters pushes that number past 140 GB, far beyond any consumer GPU and into multi-node server territory. The memory cost of model weights is the primary constraint on where LLMs can run and who can afford to run them. BitNet b1.58, introduced in 'The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits' (arXiv:2402.17764, submitted February 27, 2024), attacks that constraint at the source: instead of storing each weight as a 16-bit float, it stores each weight as one of three values: -1, 0, or 1. The name '1.58 bits' is mathematically precise, because log base 2 of 3 is approximately 1.58. The paper's central claim, verified against same-size FP16 Transformer baselines, is that ternary weights match full-precision perplexity and end-task performance while dramatically reducing latency, memory, throughput cost, and energy consumption.

The original BitNet paper and the 1-bit lineage

BitNet b1.58 builds on the original BitNet paper published October 17, 2023 (arXiv:2310.11453), also from Microsoft Research. The original BitNet used binary weights restricted to {-1, +1}, true 1-bit storage, and applied binarization only to the weight matrices in the linear layers. Activations remained in higher precision. That paper demonstrated that a 1-bit Transformer could be trained from scratch, but the binary restriction (no zero weight) meant the model had less expressive capacity per parameter than ternary alternatives. The authors of b1.58, led by first author Shuming Ma of Microsoft Research, identified that adding a third value, zero, closes most of the accuracy gap versus FP16 at the cost of only 0.58 additional bits per weight. Zero weights are particularly important because they enable the model to completely suppress a signal path without approximation, a capability binary weights can only emulate by carefully balancing positive and negative pairs.

How ternary quantization works during training

BitNet b1.58 trains the weights in full precision but quantizes them to ternary values at each forward pass during training. The quantization function maps any weight to the nearest value in {-1, 0, +1} after scaling by the mean absolute value of the weight tensor. Specifically, the weight matrix W is divided by its mean absolute value (alpha), then rounded to the nearest integer and clamped to the range -1 to 1. This quantization is applied to every linear layer weight during the forward pass, but the optimizer (AdamW in the paper's experiments) accumulates gradients in FP16 or BF16, so the model can learn to place weights near boundaries between ternary values and shift them across training. Activations use INT8 quantization with a per-token scale, keeping the inner product computation as integer multiply-and-add operations wherever hardware supports it. The combination of ternary weights and INT8 activations means that the memory-intensive matrix multiplications in attention and feed-forward layers can be recast as addition and subtraction operations on integers, bypassing floating-point hardware entirely on supported architectures.

• Each weight is stored as {-1, 0, 1} after scaling and rounding during the forward pass.
• log2(3) = approximately 1.58, giving the model its name and bit budget.
• Activations use INT8 per-token quantization; weight-activation products become integer operations.
• Optimizer state (gradients, momentum) stays in FP16 or BF16; only the forward pass uses ternary weights.
• Zero weights allow complete signal suppression, an expressiveness advantage over binary {-1, +1} weights.

Benchmark results: matching FP16 at 3B and 7B parameters

The b1.58 paper compares BitNet b1.58 models trained from scratch against LLaMA baselines of the same parameter count, trained on the same number of tokens with the same data. At 3B and 7B parameters, BitNet b1.58 matches the LLaMA baseline in perplexity on validation text from the C4 and Pile corpora. Downstream task accuracy on ARC-Easy, ARC-Challenge, Hellaswag, Winogrande, PIQA, and BoolQ follows a similar pattern: BitNet b1.58 reaches parity with FP16 LLaMA at the same parameter scale. Memory per token (the KV cache contribution) falls because the weight matrices that dominate activation memory are ternary. The paper also reports that BitNet b1.58 is Pareto-optimal compared to FP16 models: at equal memory cost, the ternary model fits a larger architecture and achieves better perplexity than the FP16 model that fits in the same memory budget.

bitnet.cpp: inference numbers on real CPUs

Microsoft Research released bitnet.cpp in October 2024 (technical report arXiv:2410.16144) as the official CPU inference framework for ternary LLMs. The framework implements hand-written kernels that pack ternary weights into 2 bits per weight using a custom I2 S format and perform the weight-activation product as table-lookup and bitwise operations rather than floating-point multiply-accumulate. On ARM CPUs, bitnet.cpp achieves speedups of 1.37x to 5.07x over equivalent FP16 inference, with larger models showing larger gains because the memory bandwidth savings compound. On x86 CPUs with AVX2 or AVX-512 instruction sets, speedups range from 2.37x to 6.17x. Energy consumption drops by 55.4% to 70.0% on ARM and by 71.9% to 82.2% on x86. The most striking benchmark in the technical report: a hypothetical 100-billion-parameter BitNet b1.58 model running on a single CPU achieves 5 to 7 tokens per second, a throughput the report describes as comparable to human reading speed. A 7B BitNet b1.58 model running on a laptop CPU with 8 threads reaches around 20 tokens per second, versus under 2 tokens per second for an FP16 LLaMA-7B on the same hardware.

The official 2B model and what the ternary roadmap looks like

Microsoft Research released the first official BitNet b1.58 model on Hugging Face in April 2025: BitNet-b1.58-2B-4T, a 2.4-billion-parameter model trained on 4 trillion tokens. It is the largest publicly released model in the BitNet family with verified training provenance from the research team. The model supports conversational use, runs on x86 and ARM CPUs through bitnet.cpp with up to 6.17x speedup over FP16 inference, and reduces energy consumption by up to 82.2% on x86. Community-trained models in the bitnet.cpp repository include a 3.3B model and a fine-tuned LLaMA-3-8B variant converted to 1.58-bit weights after pretraining on 100 billion tokens, demonstrating that post-training ternary quantization is also viable, though training from scratch in ternary produces better results at equal token budgets. In July 2025, Microsoft Research extended the BitNet approach to embedding models, releasing BitNet-embedding-0.6B and BitNet-embedding-270M, the first ternary embedding models, achieving 1.42x to 2.28x prefill speedup over FP16 on x86 CPUs with lossless embedding quality. The ternary weight direction is now active across at least four Microsoft Research product lines, suggesting that 1.58-bit storage is no longer a research curiosity but a credible path for deploying large models on CPU-class edge hardware at production scale.