On-Device AI in 2026: What Actually Ships

Until this year, every "smart" feature in a mobile app carried the same three problems: each AI request cost money, broke offline, and sent user data to someone else's server. In 2026 the default changed. Both platform vendors now ship a production LLM inside the operating system, free to call, running entirely on the phone.

This post is a state of the union from the practitioner side - what the platforms actually give you, what the numbers look like on real hardware, who has shipped it in production, and what still doesn't fit. We run these models on real devices for client work; the numbers below are either platform-official, independently reproduced, or labeled when they're vendor best-case.

What Apple gives every iOS app

The Foundation Models framework (iOS 26, expanded in 27) is a free Swift API to an on-device model in the ~3B-parameter class. It's not a raw text pipe: guided generation returns type-safe Swift structs via @Generable, and the framework handles tool calling and streaming. The hard constraint to design around is a 4,096-token context window per session - Apple throws on overflow rather than silently truncating, which is the right behavior but means context budgeting is your job.

At WWDC 2026 the model family split in two: AFM 3 Core (3B dense, now accepting image input) and Core Advanced - a 20B sparse mixture-of-experts that activates 1-4B parameters per token and needs 12GB of RAM, which today means iPhone Air / 17 Pro class hardware.

Two less-noticed pieces matter as much as the models:

• The LanguageModel protocol is one API across Apple's on-device model, Private Cloud Compute, and third-party clouds. Routing a request to a bigger model becomes a configuration change, not a rewrite. This is what makes hybrid architecture cheap to build.
• The free Private Cloud Compute tier: apps in the App Store Small Business Program (under 2M lifetime downloads) call Apple's cloud model at zero API cost. For a startup, that's a production LLM stack - device plus cloud fallback - with no inference bill at all.

For teams bringing their own models, Core AI is the successor to Core ML for generative workloads: ahead-of-time compilation, stateful execution, PyTorch import.

What Google gives every Android app

Gemini Nano runs via AICore on more than 140 million devices - Google's number, and the largest deployed on-device LLM fleet in existence. It executes inside Private Compute Core: no direct internet access, no data retention.

The practical entry point is the ML Kit GenAI APIs, now GA: summarization, proofreading, rewriting, image description, and a general prompt API, with inputs capped under 4,000 tokens and per-app quotas. At I/O 2026 Google previewed Gemini Nano 4 in two sizes - Fast (~3x faster) and Full (needs 12GB+ RAM) - plus a structured output API and prefix caching.

Two platform facts should be on every Android roadmap:

• Firebase AI Logic routing ships modes like PREFER_ON_DEVICE and ONLY_ON_DEVICE. The latter is a guarantee your data never leaves the phone - an architectural property you can put in a privacy policy.
• The AppFunctions mandate: by March 1, 2027, apps in messaging, notes, calendar, productivity, finance, and retail categories must expose AppFunctions (think of it as Android's local MCP) to comply with Google's AEP guidelines. That clock is already running.

How fast do small LLMs actually run on phones?

Decode speeds below are Q4-class quantization on flagship hardware, from platform-published or independently reproduced sources:

The sweet spot on 2026 flagships is 3B-4B. 8B runs, but you're in trade-off territory. Mid-range devices want 1B-2B or selective-activation architectures.

Two physical realities shape everything. First, the bandwidth wall: decode is memory-bound, and phones move 50-90 GB/s where a datacenter GPU moves 2-3 TB/s - a 30-50x gap that no amount of NPU marketing changes. Second, thermals: sustained inference draws 8-10W, loses 15-42% throughput within minutes, and a chassis above 45°C halves GPU clocks. Design for bursts, not marathons. (Also treat vendor NPU figures as optimized best case - independent reproductions typically land 2-2.5x lower.)

On the other side of the ledger: on-device can be faster than the cloud where it fits. Nano v3 prefill reaches ~940 tok/s on a Pixel 10 Pro, and there is no network round-trip at all.

Who has actually shipped it

The strongest public case is Kakao Mobility. They benchmarked cloud Gemini against Gemini Nano with the ML Kit Prompt API in production - a real head-to-head, not a lab demo - and chose on-device on privacy, cost, accuracy, and speed together. The business results: order completion 24% faster, new-user conversion up 45%, existing-user conversion up 6%, and more than 200% growth in AI-assisted orders during peak seasons. One scenario says it all: photos containing location data get analyzed on the device, so the sensitive data never travels.

Oura moved AI analysis of health data onto the user's phone; its Symptom Radar runs locally for speed and offline availability. (The reported ~10x cost reduction versus their previous cloud setup comes from a vendor case study - treat it as directional.) On iOS, Signeasy ships contract summarization and document Q&A through Foundation Models, and Stoic generates journaling prompts entirely on-device - the user's personal entries stay personal.

One honest note about the market: you won't find many public bank or insurance mobile on-device cases yet, because regulated verticals run their pilots quietly. The same privacy driver shows up in on-prem form - one banking-compliance deployment we've studied runs an air-gapped Llama with a vector database over regulatory circulars, cutting interpretation time from days to minutes with a full citation trail. Same architecture logic, datacenter hardware.

The economics: your AI bill stops growing with your success

On-device marginal inference cost is $0 at any volume. Apple's phrase is "zero token costs"; Google's is "no additional per-use charges." That sentence deserves a worked example.

Take a feature doing 1M short summarizations a day (500 tokens in, 100 out). On 2026 cloud pricing that's roughly $8.25K-16.5K per month, forever, scaling with usage. On-device, the same workload has zero variable cost - you paid for it once, in engineering.

How much of a real workload can move? TrafficBench, an enterprise traffic study, found 80.7% of standard enterprise LLM queries are executable on-device with sub-20B models, and a hybrid router cut total cost by 60%, GPU compute by 67%, and energy by 77% - at under 5% quality loss.

The structural point: on-device AI converts variable OPEX into a mostly fixed engineering cost. Ten users or ten million cost the same to serve.

What still doesn't fit on-device

This section is the one most vendors skip, and it's the one that saves projects. Part of our job is telling clients which use cases don't fit on the device:

• Long context. 4K tokens on both system models (32K on Tensor G5 hardware) rules out whole-document analysis of large files on most devices.
• World knowledge and deep reasoning. Device-scale models are not frontier models. Complex multi-step agentic planning (5+ tool calls), long-context reasoning, and multimodal generation stay cloud-side.
• The memory gate. The best on-device models need 12GB of RAM, which excludes a large share of the installed base - including recent devices like the Pixel 9 and iPhones below the 17 Pro/Air class.
• Sustained heavy load. Roughly 20 heavy inferences cost about 10% of a flagship battery; ~2 hours of continuous 7B generation drains it. Background batch jobs belong elsewhere.

None of this argues against on-device AI. It argues for a hybrid: everyday, high-frequency, short-context, privacy-sensitive tasks run on the phone; heavy exceptions route to the cloud through the same API. The skill is knowing where to draw the line for a specific product - that's literally the job.

A staged way to adopt it

The question in 2026 is no longer "should we try on-device AI" but "which two or three user journeys move on-device now." The staging we use with clients:

1. Pick the high-frequency, short-context features first - summarize, classify, rewrite, extract structured data from a photo. These hit the 80% bucket, benefit most from $0 marginal cost, and fit inside 4K tokens.
2. Ship with hybrid routing from day one. LanguageModel protocol on iOS, Firebase AI Logic on Android. On-device first, cloud fallback for the exceptions - a config change per feature, not an architecture rewrite.
3. Budget for the physics. Context accounting, thermal-aware scheduling, and a device-capability gate (RAM check, model-availability check) are engineering line items, not afterthoughts.
4. Measure like Kakao did. Run the on-device path against your existing cloud path on business metrics, not vibes. Sometimes the cloud wins. Often it doesn't.

We build this layer for a living - native iOS and Android, Core ML and Foundation Models, Gemini Nano integration, on-device RAG, and self-hosted LLM backends for the cloud half of the hybrid. If you're deciding where the on-device/cloud line sits in your product, that's a conversation we do well - and the first one is free and ends with a written report either way.