Small Language Models and the Move to On-Device AI

A 270-million-parameter model now runs 25 conversations on a Pixel phone for three-quarters of one percent of the battery. That single measurement, published by Google, explains why on-device AI stopped being a demo in 2025 and started being a product decision in 2026. The interesting question is no longer whether small models work, but which jobs you should stop sending to a data centre.

Why small models got good

Three things changed at once. Training recipes improved, with small models now trained on far more tokens than early scaling-law orthodoxy recommended for their size. Distillation matured into standard practice rather than a research curiosity. And architectures started being designed around a memory budget rather than a parameter count.

Google’s Gemma 3n is the clearest example of that last shift. Its developer guide, published in June 2025, describes two models whose raw parameter counts, 5B for E2B and 8B for E4B, deliberately overstate what has to fit in accelerator memory. A technique called Per-Layer Embeddings keeps each layer’s embedding parameters on the CPU, leaving roughly 2B and 4B respectively for the GPU or NPU. The result is models that run in as little as 2GB and 3GB of memory. A nested MatFormer design goes further: training the E4B model simultaneously optimises a fully functional E2B sub-model inside it, so you can extract a smaller model without a separate training run.

Quantization, explained without the linear algebra

A model’s weights are numbers. Store each one in 16 bits and a 3-billion-parameter model needs about 6GB. Store each in 4 bits and it needs about 1.5GB. Quantization is that trade: fewer bits per weight, less memory, faster arithmetic, some accuracy lost.

The engineering is in how you lose as little as possible. PyTorch’s write-up of quantized Llama 3.2 on Arm CPUs is unusually specific about the scheme: 4-bit integer weights quantized per block, 8-bit per-row activations, and scale factors kept in BF16, one per block. The measured results for Llama 3.2 1B were a model file of 1.1 GiB against 2.3 GiB for the BF16 baseline, peak runtime memory of 1.9 GiB against 3.1 GiB, and on a Samsung S24+ using six of eight CPU cores, prefill above 350 tokens per second and decode above 40 tokens per second, more than twice the BF16 decode speed and more than five times the prefill.

Two refinements matter for anyone shipping. First, quantization-aware training: instead of squashing a finished model, you train it knowing it will be squashed. Google ships QAT checkpoints for Gemma 3 270M specifically so it can run at INT4 with limited loss. Second, quantizing different parts of the model differently. Apple’s 2025 foundation-model report describes a roughly 3-billion-parameter on-device model with decoder weights at 2 bits per weight via QAT, the embedding table at 4 bits, and the key-value cache at 8 bits. It also splits the model into two blocks in a 5:3 depth ratio and shares block 2’s KV caches with the final layer of block 1, cutting KV cache memory by 37.5%.

Distillation

Distillation is the other half of the story, and the idea is old. Geoffrey Hinton, Oriol Vinyals and Jeff Dean’s 2015 paper “Distilling the Knowledge in a Neural Network” showed that a small model trained to match a large model’s full output probability distribution learns more than one trained on hard labels alone. The large model’s relative confidence across wrong answers carries information about how the task is structured. Modern small models are typically distilled from a much larger sibling on curated or synthetic data, which is why a 4B model released in 2026 has capabilities no 4B model trained from scratch on web text would have.

The models you can actually ship

Apple’s third-generation models, announced in June 2026, show where this is heading. The larger on-device model is a 20-billion-parameter sparse design that activates only 1B to 4B parameters per request, keeps the full weights in NAND flash, and swaps input-dependent routed experts into DRAM only when needed alongside a set of always-active shared experts. Parameter count is becoming a poor proxy for hardware requirements.

The silicon: what an NPU is buying you

Neural processing units are fixed-function matrix engines that do low-precision arithmetic at a fraction of the power a GPU would need. Microsoft’s Copilot+ PC developer guide sets the bar at an NPU capable of 40 or more trillion operations per second, and lists Snapdragon X Elite machines, AMD Ryzen AI 300 series and Intel Core Ultra 200V series parts as qualifying. Qualcomm’s Snapdragon X2 Elite Extreme, announced in September 2025 with devices expected in the first half of 2026, doubles that to an 80 TOPS NPU. On phones, Qualcomm claimed a 37% generation-over-generation improvement for the Hexagon NPU in Snapdragon 8 Elite Gen 5, along with INT2 support, at its September 2025 summit as summarised by Counterpoint Research.

The software side is where the NPU advantage becomes concrete. Microsoft’s documentation for Phi Silica, the local language model shipped in the Windows App SDK, is explicit that speculative decoding and prompt compression are available on the NPU path but not on the GPU fallback, which runs on GeForce RTX 30-series and newer or Radeon RX 9060-series and newer with at least 6GB of VRAM. Output quality is the same on both; throughput and power draw are not. Treat vendor TOPS figures as a ceiling, not a throughput estimate. The number is a peak arithmetic rate at some unspecified precision, and small-model decoding is usually bound by memory bandwidth instead.

What running locally actually buys you

• Latency. No network round trip, no queue, no cold start. For interactive text completion, autocorrect, summarising the message you are reading or classifying an incoming email, the round trip is often larger than the inference.
• Privacy and compliance. Data that never leaves the device does not need a processing agreement, a residency commitment or a breach-notification plan. For health, legal and Canadian public-sector procurement in particular, that changes what you can sell.
• Marginal cost of zero. Your inference bill does not scale with usage, because the user is paying for the silicon and the electricity.
• Offline. Aircraft, hospitals, mines, warehouses, rural Canada.
• Battery, measured. Google’s INT4 Gemma 3 270M used 0.75% of a Pixel 9 Pro’s battery for 25 conversations.

Where it still fails

Small models remain bad at long-horizon reasoning, broad world knowledge and anything requiring a large context window, because context is memory and memory is the constraint. They are also a distribution problem: you are shipping a multi-gigabyte binary to a fleet of devices with wildly different NPUs, thermal budgets and OS versions, and you cannot hotfix a bad output the way you can swap a server-side prompt. Quantization damage is uneven: the accuracy loss is small in aggregate and sometimes large on the one edge case your customer cares about.

How to decide what runs where

Split your feature list by whether the task is narrow, whether the input is sensitive, and whether it is called constantly. Narrow, sensitive and high-frequency work (classification, extraction, redaction, rewriting, routing, on-device search ranking) belongs on the device, and a 1B to 4B model quantized to 4 bits will usually do it. Open-ended reasoning over large context belongs in the cloud. Build the routing layer before you pick the model, because the interesting products in 2026 are the ones that run the cheap classifier locally and only escalate the 5% of requests that need a frontier model.

Sources

• Google Developers Blog — Introducing Gemma 3 270M
• Google Developers Blog — Introducing Gemma 3n: The developer guide
• PyTorch — LLM Inference for Llama 3.2 Quantized Models with ExecuTorch and KleidiAI
• Apple Machine Learning Research — Updates to Apple’s On-Device and Server Foundation Language Models
• Apple Machine Learning Research — Introducing the Third Generation of Apple’s Foundation Models
• Microsoft Learn — Copilot+ PCs developer guide
• Microsoft Learn — Get started with Phi Silica in the Windows App SDK
• arXiv — Distilling the Knowledge in a Neural Network (Hinton, Vinyals, Dean, 2015)