On-device AI: what Apple and Android model access means for apps

On-device AI runs a compact model directly on the user's phone, tablet or laptop, so the input never leaves the hardware and the first token arrives without a network round trip. In 2026 that is a real option for summarising, rewriting, classifying, extracting structured data and handling short conversational turns, but not for deep reasoning, broad world knowledge or long documents. Most products should therefore be hybrid: run locally what fits locally, escalate the rest to a cloud model, and make it obvious to the user which is which.

What on-device AI actually means in 2026

The phrase covers three distinct things, and teams get into trouble by conflating them. First, platform features that happen to run locally, such as system writing tools or notification summaries; you do not control those. Second, platform APIs that expose a system model to your app. Third, a model you compile and ship yourself with a runtime such as an edge inference library, which gives you full control and full responsibility for size, updates and device coverage.

The second option is what changed for product teams. Apple's Foundation Models framework, introduced at WWDC 2025, gives apps access to the compact model behind Apple Intelligence through a Swift API, with guided generation into typed structures, tool calling and streaming responses. On Android, the Gemini Nano class of models is reached through system-level AI services and higher-level kits that wrap common tasks such as summarising, proofreading, rewriting and image description on supported hardware. Some browsers have begun exposing built-in model APIs of their own. Names, capabilities and availability all move quickly, so treat any specific release described here as current at the time of writing and confirm against the platform documentation before you commit a roadmap.

The important shift is not the model. It is that inference has become a platform capability with an availability check, like the camera or biometrics, rather than a vendor account with a bill attached.

The four things you genuinely gain

Privacy posture. Text that never leaves the device is far easier to defend in a privacy review, a procurement questionnaire or a Canadian public sector security assessment. You are not writing a data processing agreement for a health note that a nurse dictated into a form.

Latency. Local inference removes the network entirely. For interface work such as smart replies, tagging, on-type suggestions or field extraction, the difference between a local response and a cloud round trip is the difference between a feature that feels part of the app and one that feels like a request.

Availability. Ferries, basements, job sites, aircraft, rural highways. Field apps in construction, logistics, health and utilities spend real time offline, and a feature that works there is worth more than a better answer that arrives later.

Cost shape. On-device inference carries no per-token charge, which changes what you are willing to run. You can afford to classify every message, score every form field or pre-summarise every record, because the marginal cost is battery rather than budget.

Where the small model runs out of road

A model small enough to run on a phone is not simply a shrunken frontier model. It is a different tool. Expect weaker world knowledge, shallower multi-step reasoning, a much shorter usable context, more brittle instruction following under complex prompts, and uneven quality outside its strongest languages. It also has no retrieval of its own: if the answer depends on your data, you supply that data in the prompt, and the short context is what limits you.

The practical test is not a benchmark. Take a few dozen real inputs from your own product, including the messy ones, and run them through both a local and a cloud model with the same prompt and the same output schema. Score them against what a competent human would produce. You will usually find a clean boundary: some tasks are indistinguishable, and some fall apart. Build to that boundary rather than to a general impression.

Choosing where each feature runs

Designing the hybrid path properly

A good hybrid design is a routing decision made once, in code, with clear triggers. In practice four triggers cover most products: the input exceeds the local context budget; the local output fails schema validation or a confidence check; the task is on your declared cloud-only list, such as long document analysis; or the user explicitly asks for a stronger answer. Everything else stays local.

Keep the contract identical on both sides. Define one output schema, validate against it regardless of which model produced the result, and log the route taken rather than the content. That gives you an escalation rate you can actually watch: escalations divided by total requests, split by trigger. If that number drifts upward, either your inputs have changed or your local prompt has decayed, and both are worth knowing before your invoice tells you.

A hybrid feature is only as trustworthy as its quietest moment. If a user cannot tell whether their words stayed on the device, you have not shipped a privacy feature.

Make escalation visible and, where the content is sensitive, make it a choice rather than a default. A short, honest control beats a paragraph in a policy page. And always keep a third path: when no model is available, the feature should degrade to something useful, such as a template, a manual form or a plain search, instead of an error state.

Device floors, OS floors and honest availability checks

Do not gate your whole app on the OS version that carries the model. Gate the feature. The pattern that holds up is a runtime availability check with three outcomes: the local model is ready, it is supported but not yet downloaded or currently unavailable because of storage, battery or thermal state, or it is unsupported on this hardware. Each of those deserves different copy and a different fallback, and the middle case is the one teams forget.

Avoid hard-coded device lists. They go stale quickly, and they are wrong in both directions. Ask the platform. Then decide your floor commercially rather than technically: if a meaningful share of your users sit below it, either the feature ships cloud-only for them or it waits. On iOS the floor tends to be recent hardware plus a current OS; on Android the picture is more varied across manufacturers, so plan for a wider spread of outcomes and test on mid-range devices rather than flagships. If you ship your own weights instead, budget for app size, first-run download, storage pressure and the support load that comes with all three.

What to build differently this year

Three habits separate teams that get value from on-device AI from teams that get a demo.

• Treat structured output as the interface. Guided generation into typed structures is the single biggest reliability gain available on small models. Ask for a schema, validate it, and let the failure be a retry or an escalation rather than a paragraph of prose your code has to parse.
• Build the evaluation harness first. A fixed set of real inputs, expected outputs and a scoring rule, run in continuous integration. The model can change under you when the OS updates, so a regression suite is not optional maintenance; it is the only warning you will get.
• Design the interface for uncertainty. Stream partial output, allow interruption, show what is editable, and never present a generated value as a committed fact. Meeting WCAG 2.2 AA matters here too: streaming text, live regions and dynamic controls need announcements and focus handling that a static screen does not.

One more practical note for Canadian products: bilingual quality is not uniform across small models. If you ship English and French, evaluate both separately and be prepared to route French requests to the cloud even when English stays local.

How OlDevs helps

OlDevs is a full-stack technology studio in Vancouver, building software since 2014. We take on-device and hybrid AI work the way we take on everything else: one accountable team, a working demo every week, and full ownership for you of all code, designs, accounts and intellectual property. Our AI development practice covers the parts that decide whether this works in production, including task boundary testing, prompt and schema design, evaluation harnesses, routing logic, and the privacy disclosures that go with it. We work remotely with clients across Canada, on video calls in your time zone, with on-site visits when the work calls for it, and we deliver in English and French.

If you are weighing a local-first feature, an OS floor decision or a hybrid architecture and want a straight assessment rather than a pitch, tell us what you are building. We reply to every enquiry within one business day. Request a quote and we will start with the boundary question: what actually has to stay on the device.