Streaming a 6.7-billion-parameter model’s weights off phone flash costs about three seconds per token. Not per response. Per token. The number comes from Kim, Han and Kim’s LLM-on-the-Palm (ICCAD 2025). In their NPU-only baseline, weight I/O accounted for 89.5% of energy. The chip was never the expensive part.
Autoregressive decoding at batch size one reads essentially every parameter to produce every single token, which makes throughput a bandwidth problem rather than a FLOPs problem. The NPU rating on the spec sheet — 45, 48, 50 TOPS — measures a resource that is not the bottleneck. Everything else about on-device models follows from that one fact: why quantization is mandatory rather than optional, why the accuracy cliff sits where it does, and why two runtimes at the same bit-width can differ tenfold in energy per token.
The same paper gives the layout plainly. LPDDR5 delivers roughly 50 GB/s, but NAND flash — where the weights must live, since a 16-bit 6.7B model needs more than 13 GB against a phone’s 6–14 GB of DRAM — delivers about 4 GB/s. Their processing-in-NAND architecture brings the per-token cost down to roughly 100–120 ms, and it gets there by attacking the traffic rather than the math.
The pattern is not specific to phones. Pope and colleagues at Google, in Efficiently Scaling Transformer Inference (MLSys 2023), locate the constraint in the memory traffic needed to load parameters and KV cache from memory at every decode step. Gholami and co-authors put numbers on the divergence in AI and Memory Wall (IEEE Micro, 2024): across roughly two decades, peak server compute grew about 60,000× while DRAM bandwidth grew about 100× — 3.0× versus 1.6× per two-year period. Their sharpest illustration is that GPT-2 runs slower than BERT at equal FLOPs, because decoder-only generation is a matrix-vector operation with low arithmetic intensity rather than the matrix-matrix work an accelerator is built for.
This is why quantization is not an optimization. It is the entry fee. Halving the bit-width halves the bytes moved per token and therefore roughly doubles the ceiling. It is the only lever with that property.
Dettmers and Zettlemoyer established the shape empirically in The case for 4-bit precision, across more than 35,000 zero-shot experiments from 19M to 176B parameters: 4-bit gives the best accuracy-per-bit trade-off, and performance degrades below it, consistently across scales.
The peer-reviewed picture since has sharpened one distinction that matters enormously in practice — the difference between quantizing weights only and quantizing activations too. A systematic review of 52 post-training quantization studies by Czakó, Kertész and Szénási (IEEE Access, 2025) finds 8-bit weights-and-activations essentially free against FP16, and 4-bit weight-only routine. Push activations to 4 bits as well and the same group’s measurements (IEEE SMC, 2025) show LLaMA-2 7B falling from 69.6% average zero-shot accuracy to 55.9–59.8%, with WikiText-2 perplexity going from 5.47 to 7.51–8.33. LLaMA-3.1 8B drops from 73.1% to 60.7–62.9%. Mistral 7B, more gracefully, 73.1% to 66.9–67.5%.
The step below is not a step. Xu and colleagues (IEEE CloudCom, 2025) report LLaMA-1 7B at W3A3 collapsing from 62.2% accuracy and 6.4 perplexity to roughly 35–36% accuracy and perplexity in the range of 19,000–30,000. Their own low-rank error-correction scheme recovers perplexity only to about 47–76 — better by three orders of magnitude, still unusable. The mechanism the review identifies is specific: activation outliers exceeding a thousand times the median hidden-state magnitude inflate the quantization step so much that ordinary values underflow to zero.
So the working default is 4-bit weight-only, which is close to free, and 4-bit activations is a deliberate trade carrying a ten-point tag. Below that, post-training quantization stops being the right tool at all.
Here is the finding I did not expect, and the one with the most direct engineering consequence. Rajput and Sharma (IEEE ICSA-C, 2024) quantized LLaMA-2 7B to 4 bits five different ways and measured both quality and energy. Perplexity varied moderately — 5.96 for GGUF, 6.02 AWQ, 6.09 GPTQ, 7.54 GGML, 7.90 BitsAndBytes. Energy did not vary moderately. The same benchmark cost 308–318 mWh under GGML/GGUF versus 528 for BitsAndBytes, 809 for AWQ, and 1,123 for GPTQ. Throughput ranged from 24–51 tokens/second at the GPTQ/AWQ end to 290–342 at the GGML/GGUF end. Tokens per milliwatt-hour spanned 0.37 to 3.42 — roughly tenfold, at constant bit-width.
I would treat the absolute milliwatt-hours as specific to that harness and hardware. The ratio is the durable finding: bit-width sets how many bytes must move, but the kernel decides what moving them costs. Anyone choosing a quantization format on perplexity tables alone is optimizing the smaller of the two variances. Pick the runtime first, then the bit-width.
Per-token energy in the wild is worse-characterized than it should be. The cleanest phone measurement I found is PowerInfer-2 (arXiv:2406.06282) at 0.257 J/token for Bamboo-7B on a OnePlus 12 — though that is the authors’ own tuned configuration, reported as beating QNN and llama.cpp on the same handset by 31% and 62%, so read it as a good-case result rather than a representative one. On an A100, Nik, Riegler and Halvorsen (Scientific Reports, 2025) metered Qwen2.5-7B at 0.016–0.028 Wh per 100-token query, and found decoding strategy alone changing energy by up to 2× at unchanged output quality. Meanwhile a TDP-derived CPU study (Gaddam & Kulkarni, AIxSET 2024) puts small-model inference at 160–590 J/token — three orders of magnitude above the metered figures, because it estimates from CPU time × thermal design power rather than measuring the wall. Both can be honestly reported; they are not measuring the same thing. Treat any unattributed joules-per-token number as decorative.
The capability question has a cleaner answer than the discourse suggests. Garg and colleagues (IEEE FLLM, 2025) fine-tuned models at 250M parameters or below and benchmarked them against Mistral-7B, Qwen3-235B-A22B and DeepSeek-R1-671B. The small models matched or beat all three on four of six tasks — sentiment analysis 95–96% against 68–94%, natural language inference 88.0–88.4% against 56–80%, code summarization BLEU 24–26 against 4–19, open-ended content creation at BERTScore ≈0.79 against 0.80–0.81 — at inference emissions the authors put as much as 13,000× lower per query. On the other two they were not close: HumanEval pass@1 of 12.8% against 36–66%, and reasoning accuracy 47% against 64–69%.
The same boundary shows up in self-training. Costello and colleagues’ Think-Prune-Train (IEEE ICLAD, 2025) lifts Gemma2-2B on GSM8K from 41.9% to 57.6%, and Gemma2-9B to 82% — above LLaMA-3.1-70B’s 78% on that benchmark. On CodeContests the same procedure moves 2B from 0.90% to 1.14%. Gains concentrate exactly where ground truth is machine-checkable, which is a statement about the training signal, not about the models.
That gives a scoping rule with evidence under it: put classification, extraction, rewriting and summarization on-device, and route open-ended synthesis somewhere else. It is the same split I argued for more generally about industrial-grade agents, and it is what the shipping products already do.
Small-model scores need a larger haircut than large-model scores, and there is now direct evidence for that asymmetry. Masry, Ahmed and Mahbub (IEEE MLSP, 2025) perturbed vision-language benchmarks and found the 2–4B open models — Qwen2-VL-2B, Phi-3.5-Vision-4B — the most contaminated in the pool: about 64% contamination on DocVQA and 51% on ChartQA under question modification, still emitting the memorized answer after the question had been altered so that answer was wrong, and still answering correctly when the image was swapped for an unrelated one. Closed models scored 13–46%.
On the text side, the GSM1k study (Zhang et al., arXiv:2405.00332, NeurIPS 2024) built a fresh GSM8k-style benchmark and found accuracy drops of up to 8%, naming the Phi and Mistral families as consistent overfitters, with a Spearman’s r² of 0.36 between a model’s tendency to regurgitate a GSM8k item verbatim and its GSM8k-to-GSM1k gap. Alongside that, the Phi-4 technical report (arXiv:2412.08905) documents decontamination by 13-gram and 7-gram string matching, while Yang and colleagues (arXiv:2311.04850) show n-gram decontamination is defeated by paraphrasing the test data. Those two facts sit in tension; I am not claiming the second invalidates the first, only that the standard method has a known bypass and vendors still use it. My earlier piece on measuring progress toward AGI covers why contaminated instruments are the field’s most under-priced measurement error.
Apple’s 2025 foundation models report is the most specific vendor disclosure available: roughly 3B parameters on-device, decoder weights at 2 bits per weight via quantization-aware training, 4-bit embeddings, 8-bit KV cache, and KV-cache sharing across blocks that Apple claims cuts KV memory by 37.5%. Two bits per weight is below where post-training quantization collapses, and that is the whole point of doing it during training rather than after. It is also the answer to the question I left open earlier: below 3 bits, quantize in training or not at all. Anything harder routes to a larger server model under Private Cloud Compute; Apple does not publish the routing rule. Google’s Gemini 1.0 report specified Nano-1 at 1.8B and Nano-2 at 3.25B, both 4-bit; current-generation figures I could not pin to a primary source, so I won’t quote one. Microsoft has disclosed neither parameter count nor bit-width for Phi Silica. Qualcomm reports over 350 tokens/second prefill for Llama 3.2 1B/3B on a Galaxy S24+ — a vendor benchmark, and prefill is the compute-bound phase, not the bandwidth-bound one.
Which brings this back to the spec sheet. AMD rates XDNA 2 at 50 TOPS; Qualcomm and Intel advertise roughly 45 and 48 for their current NPUs. Those are real numbers about the wrong resource. They describe the prefill phase — reading your prompt — and say nothing about the decode phase, which is where the user waits. Measured on a mid-range phone without a dedicated NPU, Joshi and colleagues (IEEE ICETI4T, 2025) got 3.5–8.5 tokens/second from LLaMA 3.2 1B and 3.9–15.2 from Gemma2 2B under Ollama with Q4 GGUF — well under the ceiling a 50 GB/s bus would imply, though the review does not report that handset’s actual bandwidth, so I would not compute a precise efficiency from it.
In 1989 Intel shipped the 80170NX, the ETANN — Electrically Trainable Analog Neural Network — with 64 analog neurons and 10,240 floating-gate synapses on a 1µm CHMOS-III process, computing dot products in the analog domain at a reported ~2,000 million connections per second. Dedicated local neural silicon, no host CPU in the inner loop, thirty-five years before “NPU” reached a laptop spec sheet. It was characterized in SPIE Proceedings 1965 (1993) and independently evaluated in Fermilab TM-1798. It failed commercially — training was awkward, precision poor, and the data was two decades away. The idea was fine. The memory system it would have needed did not exist, which is, more or less, still the problem.
References