What a Tenth of a Bit Buys

The LittleBit repository says it is “designed for 1.0 to 0.1 bits per weight.” Taken at face value, 0.1 bits against the 16 bits of an ordinary half-precision weight is a 160-fold cut in storage. That is the number a reader will carry away from a headline. It is also the wrong number to carry into a gigabyte budget, because the paper counts bits one linear layer at a time and a model’s memory footprint counts everything. The sections below cover the count, the footprint, and the quality curve.

How the bits are counted

Each linear layer of shape \(d_{out} \times d_{in}\) is replaced by a product of two binary factors, scaled by a handful of 16-bit vectors. The rank \(r\) of the factorization is chosen to hit a target budget. Appendix D of the paper gives the bookkeeping:

\[b = \frac{2r(d_{out}+d_{in}) + 32(d_{out}+d_{in}) + 32r}{d_{out}\,d_{in}}\]

The leading 2 is for the residual path, which the method uses for error compensation. The binary factors cost one bit per element. The three scale vectors cost 16 bits per element. The paper’s worked example for a \(4096 \times 11008\) layer sets \(r = 133\), which gives \(b \approx 0.0999\). Everything inside that layer, signs and scales and the residual path, is inside the 0.1. Nothing outside the layer is.

The paper also does not describe a codebook. The bits that matter are the ones in the factorization itself.

What the footprint says

The paper’s Table 3 reports whole-model memory for the Llama2 family. The numbers are the ones a reader can check against a model card:

The gap between the 160× nominal figure and these is not a rounding issue. The table’s caption says quantization applies only to the Transformer blocks, “while components like the embedding layer and the lm_head remain in FP16.” Those parts are a fixed cost the 0.1 figure cannot touch. The paper does not break them out, so I cannot say how large they are for each model. The reduction climbs from 21× to 70× as the model grows, which fits a fixed slice of the footprint that weighs less on larger models.

Nominal and realized memory reduction at 0.1 bits per weight For three Llama2 models, a dumbbell runs from the nominal 160 times reduction (16 bits divided by 0.1) to the realized whole-model reduction in Table 3 of the paper: 21.4 times for Llama2-7B, 31.0 times for Llama2-13B, and 69.7 times for Llama2-70B. MEMORY REDUCTION · WHOLE MODEL · LLAMA2 hollow: nominal 16 ÷ 0.1 · filled: realized 0× 40× 80× 120× 160× Llama2-7B Llama2-13B Llama2-70B 160× — 16 bits ÷ 0.1 bit per weight, nominal 160× — 16 bits ÷ 0.1 bit per weight, nominal 160× — 16 bits ÷ 0.1 bit per weight, nominal 21.4× — FP16 13.49 GB ÷ 0.63 GB, Table 3, arXiv 2506.13771v5 (5 Feb 2026) 31.0× — FP16 26.06 GB ÷ 0.84 GB, Table 3, arXiv 2506.13771v5 (5 Feb 2026) 69.7× — FP16 138.04 GB ÷ 1.98 GB, Table 3, arXiv 2506.13771v5 (5 Feb 2026) 21.4× 31.0× 69.7× 160× 160× 160× Realized = FP16 GB ÷ LittleBit GB at 0.1 BPW. Embeddings and lm_head stay FP16.
The headline 160× assumes every byte of the model is quantized. The paper's own footprint table gives 21× to 70×, depending on size.

What the quality costs

The footprint is only half the trade. The paper’s Table 1 reports WikiText-2 perplexity across model sizes and bit budgets. On Llama2-7B the FP16 model scores 5.47. LittleBit scores 9.08 at 1.0 bits per weight, 10.47 at 0.55, and 15.92 at 0.1. The last of those is about 2.9 times the FP16 perplexity.

The 1.0-bit row is the honest comparison, because the paper evaluates its method against other 1-bit baselines. On the same model, OneBit scores 8.36 and BinaryMoS scores 7.74. LittleBit trails both. It does beat BiLLM at 1.1 bits, which scores 29.00. The 0.1 setting is where the cost against full precision is largest.

Zero-shot accuracy tells the same story. Table 2 reports an average of 62.97 for FP16 Llama2-7B across its common-sense benchmarks, and 47.26 for LittleBit at 0.55 bits, about 15.7 points lower. The paper’s Table 2 compares LittleBit against STBLLM at matched budgets and wins on that comparison. Table 2 stops at 0.3 bits, so I make no claim about zero-shot accuracy at 0.1.

WikiText-2 perplexity for Llama2-7B across bit budgets Dot plot of WikiText-2 perplexity on Llama2-7B, lower is better: FP16 at 5.47; BinaryMoS at 1 bit, 7.74; OneBit at 1 bit, 8.36; LittleBit at 1.0 bit, 9.08; LittleBit at 0.55 bit, 10.47; LittleBit at 0.10 bit, 15.92. PERPLEXITY · LLAMA2-7B · WIKITEXT-2 · LOWER IS BETTER 6 8 10 12 14 16 FP16 (16 bpw) BinaryMoS (1 bpw) OneBit (1 bpw) LittleBit (1.0) LittleBit (0.55) LittleBit (0.10) 5.47 — FP16, Table 1, arXiv 2506.13771v5 7.74 — BinaryMoS at 1 bit, Table 1, arXiv 2506.13771v5 8.36 — OneBit at 1 bit, Table 1, arXiv 2506.13771v5 9.08 — LittleBit at 1.0 bit, Table 1, arXiv 2506.13771v5 10.47 — LittleBit at 0.55 bit, Table 1, arXiv 2506.13771v5 15.92 — LittleBit at 0.10 bit, Table 1, arXiv 2506.13771v5 5.47 7.74 8.36 9.08 10.47 15.92 Baselines at 1 bit per weight on the same model. Table 1 of arXiv 2506.13771v5.
At 0.1 bits the perplexity is about 2.9 times the FP16 score. Lower budgets buy a smaller footprint and a steeper quality price.

Which version I read

The arXiv record has five versions. The first is from 30 May 2025, and the paper was accepted to NeurIPS 2025. The version I read in full is v5, dated 5 February 2026. The bits formula in Appendix D and the figures in Tables 1 to 3 are checked against that version. I did not compare v5 against the earlier versions, so I cannot say which figures moved between them.

The repository’s history runs from a 24 November 2025 initial commit to code commits in March and May 2026. Its README describes a follow-up, LittleBit-2, which changes the initialization, and is listed as ICML 2026. I read only the LittleBit-2 abstract, which claims a state of the art in the sub-1-bit range. I have not checked its tables, so I say nothing about its numbers here.

What the number measures

The 0.1 is a property of one matrix at a chosen rank. The memory claim is about a whole model, and for the 7B case it comes out to 21 times, not 160. The quality claim is about a perplexity score on a benchmark, and at that budget the cost is roughly a threefold rise in perplexity. Those are three different measurements, and the headline sits on the first.

For the size side of this question, see the earlier essay on small models on device. The check here is the same: look at the denominator before the ratio.

References

  1. Lee, B., Kim, D., You, Y., Kim, Y. (2026). LittleBit: Ultra Low-Bit Quantization via Latent Factorization. arXiv:2506.13771v5, 5 February 2026. Full text at arxiv.org/html/2506.13771v5. Source for Appendix D, Tables 1–3.
  2. Lee, B., Kim, D., You, Y., Kim, Y. (2025). The same paper, first version dated 30 May 2025; accepted to NeurIPS 2025 per the arXiv comments field. Background only.
  3. SamsungLabs/LittleBit. Repository README and commit history, 24 November 2025 to 6 May 2026. Licence CC BY-NC 4.0.
  4. Lee, B., Kim, Y. (2026). LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment. arXiv:2603.00042, first posted 9 February 2026. Abstract read only; tables not checked.