What On-Device AI Still Can't Do (And Why That's Fine)

Part of our job is telling clients which use cases don't fit on-device - it saves projects more reliably than any optimization. The pattern we want to prevent is always the same: a team sees the demo, moves a flagship feature onto a local model, and discovers the limits in production instead of in design review.

So here is the honest list, with numbers. If you've read the state of the union, this is the sober appendix it deserves.

Limit 1: the context window is small

The system models cap at 4,096 tokens per session - both Apple's Foundation Models and the ML Kit GenAI APIs (32K exists on Tensor G5 hardware, which is a small slice of the fleet). Apple's framework throws on overflow rather than silently truncating: correct behavior, but it means context budgeting is your engineering problem.

What that rules out: whole-contract analysis, long-conversation memory, multi-document synthesis. What fits comfortably: a message thread, a receipt, a page of text, a structured record - which is most of what high-frequency product features actually process.

Design consequence: features that need long context are cloud features. Don't fight it with heroic chunking on a phone; route them.

Limit 2: the memory gate excludes real devices

The strongest on-device models want 12GB of RAM - iPhone Air / 17 Pro class, or Android flagships. That excludes a large share of the installed base, including recent hardware like the Pixel 9 and base iPhones below the 17 Pro line. And RAM pressure compounds: a 3B model plus a long KV cache can exhaust an 8-12GB device that also wants to keep your app and the OS alive.

Design consequence: on-device AI is a progressive enhancement, not a baseline. You need a capability check at runtime and a defined fallback - smaller local model, cloud call, or graceful feature absence. Which fallback you pick is a product decision (and for sensitive data, a privacy decision), not an afterthought.

Limit 3: physics - thermals, battery, bandwidth

Three numbers to design against:

• Sustained inference draws 8-10W and loses 15-42% throughput within minutes; a chassis above 45°C halves GPU clocks.
• Roughly 20 heavy inferences cost about 10% of a flagship battery; continuous 7B-class generation drains a full charge in ~2 hours.
• Decode speed is memory-bandwidth-bound: phones move 50-90 GB/s against a datacenter GPU's 2-3 TB/s - a 30-50x gap that no NPU marketing changes. Real-world 7B decode lands around 8-15 tok/s.

Design consequence: on-device is built for frequent short bursts - summarize, classify, extract, rewrite - not marathons. Batch jobs, long generations, and background churn belong in the cloud (or, minimum, behind a charger-connected check). Also budget skepticism for vendor NPU benchmarks: independent reproductions typically land 2-2.5x below the slideware number.

Limit 4: the quality gap is real for the hard 20%

Device-scale models are not frontier models, and the gap shows exactly where you'd expect: broad world knowledge, multi-step agentic planning beyond ~5 tool calls, long-context reasoning, and multimodal generation. Enterprise traffic studies put about 80% of typical LLM queries within on-device capability - which means about 20% genuinely is not, and that 20% often includes your most visible, most marketed feature.

Design consequence: benchmark on your workload before believing either direction. Kakao Mobility's on-device path beat their cloud path on accuracy - for their structured, short-context tasks. Your long-form generation feature might lose badly. Both results are normal.

Why none of this is an argument against on-device

Look at the four limits together and notice what they describe: the boundary of a well-defined region. Short context, high frequency, bursty, latency- and privacy-sensitive - inside that region, on-device inference is faster than the cloud, costs $0 marginal, and keeps data on the phone. Outside it, the cloud earns its bill.

That is not a compromise architecture; it is just correct routing. The platforms agree - Apple's LanguageModel protocol and Android's Firebase AI Logic exist precisely to make on-device/cloud routing a configuration decision instead of two codebases. A hybrid router over enterprise-shaped traffic measures out at about 60% total cost reduction at under 5% quality loss.

The teams that get burned are not the ones that use the cloud too much. They are the ones that didn't decide - that let each feature land wherever the demo happened to run, and found the limits one incident at a time.

The checklist we run before any on-device commitment

1. Does the feature's context fit in 4K tokens, with margin?
2. What share of the real device fleet passes the RAM/model-availability check, and what exactly happens on the devices that don't?
3. Is the usage pattern bursty or sustained? What is the battery budget per session?
4. Has the on-device path been benchmarked against the cloud path on business metrics, on this workload?
5. Which data classes does the feature touch, and does the fallback path respect them?

Five questions, answerable in a short technical engagement, and they determine whether a feature ships on-device confidently or routes to the cloud deliberately. If you want them answered for your product, that's a conversation we do well - free, and you keep the written assessment either way.