Local LLMs on Mobile Devices: A Developer's Field Guide

A local LLM is a language model, the kind of thing ChatGPT or Claude are built on, that runs directly on the user's phone. No server call. Two years ago I would have told you this was a research demo. The models that produced useful answers were tens of gigabytes, phones don't have data-center GPUs, and the math just didn't work.

Then the math changed. Modern Android phones ship with an NPU (Neural Processing Unit). Modern iPhones have the Apple Neural Engine. Both are chips built specifically for running ML workloads on a battery. At the same time, the open-source world got better at shrinking models without breaking them.

The combined result is easiest to show with a single before-and-after. On a Galaxy S24 Ultra two years ago, Gemma 3 1B needed 3.8 GB of disk, 4.1 GB of RAM, and took 5.6 seconds to start replying. The same model on the same phone today, in a 4-bit build: 657 MB on disk, under 1 GB of RAM, about two seconds to first word.

That gap is why mobile teams stopped asking can we? and started asking which one, on which devices, and what does it cost the user?

When does running the model on-device actually make sense?

Local inference wins on first-token latency, privacy, per-request cost, and offline behaviour. It loses on raw capability and on how heavy your app becomes. The trade is per feature, not per app. Almost every production stack I've looked at ends up mixing both.

Ship a local model when the feature is narrow, needs privacy, or has to work offline. Ship a cloud model when you need open-ended reasoning, or when your users will tolerate the network round-trip. Hybrid stacks (local for the quick answers, cloud for the long ones) are getting common, and they're usually the right answer.

If your feature works fine over a 4G connection, the cloud is probably the cheaper choice for both you and the user's battery.

What "model size" really means on a phone

If you've only built web or backend apps before, the number you usually hear about a model is its parameter count: "7B model," "70B model." On a phone, parameter count is half the story. Parameters are the numbers the model learned during training; a "1B" model has a billion of them. More parameters means smarter answers, more bytes on disk, more compute per generated word. So far, so familiar.

What's different on a phone is quantization: the number of bits used to store each parameter at runtime. Training uses 32 bits per number. For running the model (not training it), researchers worked out that most of those bits can go without much quality loss. 16 bits is easy. 8 bits is mostly easy. 4 bits is where the interesting work is happening.

Modern 4-bit techniques like SpinQuant (from Meta) and QAT+LoRA preserve roughly 92–99% of the original model's behaviour, depending on the technique. Meta's own measurements show QAT+LoRA retaining about 98% of BF16 accuracy on Llama 3.2 1B, and SpinQuant retaining about 92%. All while cutting the file and the RAM proportionally.

The same model in two precision formats is genuinely a different deployment. Different size, different speed, slightly different quality.

RAM usage at runtime is always larger than the file on disk. The runtime needs working memory for the conversation context, intermediate calculations, and the inference engine itself. Plan for 1.3 to 1.7× the file size on disk.

Two more terms come up everywhere. TTFT is time-to-first-token: how long the user waits before any text appears. Decode speed is how fast the rest streams, measured in tokens per second. A "token" is roughly half an English word, so 20 tokens/second is around 10 words/second.

So when someone tells you "Llama 3 is too big for phones," ask which build they mean. The model and the build are different things, and you ship the build.

Which phones can actually run a model?

The Android landscape today splits roughly three ways, and the model file on disk maps almost directly onto which tier can host it.

• Mid-range Android has 4 to 6 GB of RAM and a Snapdragon 7-series chip or equivalent. Pixel 7a, Galaxy A55, OnePlus Nord. This is where most of the install base actually lives, and most teams underestimate it.
• Recent flagships have 8 to 12 GB of RAM and a top chip from the last two generations. iPhone 15 Pro, Galaxy S22 and newer, Pixel 8, OnePlus 12.
• Top of the line has 12 to 16 GB of RAM and the very latest chip. iPhone 15 Pro Max, Galaxy S24+/S25 Ultra, OnePlus 12 with 16 GB.

Under 1 GB. Runs almost anywhere. Gemma 3 1B at INT4 (~657 MB), Qwen2.5-0.5B at INT8 (~521 MB), and Whisper large-v3-turbo (~626 MB in WhisperKit's Core ML build) all live in this bucket. They work on phones with 4–6 GB of RAM if you're careful with KV cache and don't try to keep two models loaded at once. This is the right starting point if your user base includes mid-range Android phones.

1–3 GB. Recent flagships only. Llama 3.2 3B at 4-bit (~2.4 GB), Gemma 3n E2B (~3 GB), and Qwen2.5-1.5B at INT8 (~1.6 GB) sit here. You're targeting devices with 8+ GB of RAM and a competent NPU: iPhone 15 Pro and newer, Galaxy S22 and newer, OnePlus 12. This is where most production "local assistant" features run today.

Over 3 GB. Top-of-the-line only. Gemma 3n E4B (~4.2 GB) and Llama 3.1 8B at 4-bit (~4–5 GB) behave well only on phones with 12 to 16 GB of RAM. Meta noted in their ExecuTorch docs that the unquantized Llama 3.2 3B reference build was tested on a single phone: a OnePlus 12 with 16 GB of RAM.

Anything older than a Pixel 6 or iPhone 13 gets the under-1-GB bucket, and even that grudgingly. Older phones don't just have less RAM. They have weaker NPUs and worse thermal headroom, so a model that runs at 15 tokens/second on a new phone might crawl at 3 on a four-year-old one. The cleanest path is to bundle a small model everywhere and gate anything larger behind a runtime device check.

What does it cost the user? Battery, thermals, and storage

Run on the NPU, or on Apple's Neural Engine. Never on the CPU. The dedicated silicon costs roughly an order of magnitude less energy for the same computation. CPU-only inference is a debugging path, not a production one.

The phone will throttle itself on a long generation. A model that produces 30 tokens/second when the chip is cool can drop to 10 once it warms up. If your UX assumes fast streaming, design for the slow case: show progress, let the user cancel, do not autoscroll past where the cursor actually is.

Storage is the quieter cost. A 4 GB model is 4 GB of the user's storage they did not ask for, and they will notice when they go to install something else and the phone says no. Both app stores expect large model files to be downloaded after install, not bundled into it. Make the user open the app, see what the feature does, and consent to the download with a progress bar. Users are already used to this from games.

The model file is dead weight in your user's storage until they use the feature. Treat it like a paid expansion, not a build asset.

Do you need to follow the EU AI Act?

If your app is available in the EU, yes. Even if the model never leaves the phone. The Act regulates placing AI on the market, not where it runs. The good news for most consumer apps is that the obligations are lighter than the headlines suggested when the Act was being negotiated.

The EU AI Act entered into force on August 1, 2024, with the rules phased in across two years:

• February 2025: prohibitions on a small list of forbidden uses (social scoring, dark-pattern manipulation, untargeted face scraping).
• August 2, 2025: obligations for providers of large general-purpose models. This hits OpenAI, Anthropic, and Google. Not most app developers.
• August 2, 2026: most remaining obligations, including the transparency rules in Article 50.

Uses are sorted into four risk tiers. Most consumer mobile features land in limited risk, which requires one thing: transparency. If a user is interacting with an AI, they have to be told. If a piece of content was AI-generated or AI-edited, that has to be clear. In practice this is a UI affordance (a label, a disclosure on first run, a watermark on a generated image), not a regulatory filing.

High risk is reserved for the uses listed in Annex III of the Act: biometric identification, employment decisions, access to essential services like credit and healthcare. If your on-device model is doing one of those, you'll need conformity assessment and registration. Talk to a lawyer before you ship.

GDPR is still in effect, separately from the AI Act. Local inference is a privacy win (the data does not cross the network). But if you collect prompts, completions, or telemetry back to your own server, the same GDPR rules apply as for any other processing. Document what you collect, pick a lawful basis under Article 6, write a clear privacy policy.

For the model file download itself, no EU law specifically requires consent to store a 2 GB blob on a user's device. But:

• The ePrivacy Directive governs storage of information on user devices. It's the legal basis behind the cookie banners, and a multi-gigabyte model file is comfortably within scope.
• Apple's App Store Review Guidelines and Google Play's Asset Delivery policies both expect transparent prompts for significant cellular downloads.
• It's good product practice anyway. A "Download local AI model (~2 GB, Wi-Fi recommended)?" dialog handles the legal corner case and a thousand support tickets in one go.

Write a privacy policy, add a "this is AI" disclosure in the UI, ask before downloading the model file. That covers the realistic compliance surface for roughly 95% of consumer features.

How to choose a model

Five questions, in order. The first one that constrains you wins.

1. What is the task?

• Text chat, summarization, instruction following → a general LLM (Gemma, Llama, Qwen).
• Speech-to-text or live transcription → Whisper large-v3-turbo.
• Tagging what's in a camera frame → MobileNetV4 (classification) or EfficientDet-Lite0 (object detection).
• Text + image + audio in one model → Gemma 3n.

2. What is your size budget? Driven by your minimum supported device. Under 1 GB if you support mid-range Android. 1–3 GB if you're recent-flagship-only. Over 3 GB only for top-of-the-line phones.

3. What license can you ship? This kills more candidates than capability does.

• Apache 2.0 (most Qwen 2.5 sizes including the 0.5B and 1.5B builds, MobileNetV4): the most permissive. Anything goes. The 3B and 72B Qwen sizes are under a separate Qwen license; check before you ship those specific variants.
• Llama Community License: fine for almost everyone, but you can't ship past 700 million monthly active users without renegotiating.
• Gemma Terms of Use: commercially usable, but you have to accept Google's terms and pass them through to your users.
• AGPL 3.0 (YOLO11): a serious problem for closed-source apps. Avoid unless legal has already cleared it.

4. What is your minimum supported phone? Set it once and design backwards. If you support phones with 4 GB of RAM, you're under 1 GB and probably INT8.

5. Which runtime are you ready to integrate? You don't write the inference engine yourself. You pick one that knows how to load the model. Each model family has a preferred path.

• LiteRT / MediaPipe: Google's official on-device runtime, the successor to TensorFlow Lite. Use it for Gemma, Gemma 3n, EfficientDet, and the LiteRT builds of Qwen. Android and iOS sample apps are both maintained.
• ExecuTorch: PyTorch's mobile runtime, Meta's official path for Llama on phones. Strong on both Android and iOS.
• WhisperKit: a Swift package from Argmax for running Whisper on iPhone. The cleanest route if you're iOS-first and need speech-to-text.
• Qualcomm AI Hub: Snapdragon-tuned builds of popular models. Worth knowing if your install base is Android-heavy.
• Core ML: Apple's framework for running models through the Neural Engine. Most other models can be converted into Core ML format.
• ONNX Runtime: cross-platform fallback. Use it for models without a vendor-blessed mobile runtime (Phi-3, custom-trained models).

Three concrete shortlists by use case:

• Privacy-first chat assistant: Gemma 3 1B at INT4 (under 1 GB). Step up to Llama 3.2 3B SpinQuant (~2.4 GB) if you can. Only ship Gemma 3n E4B or Llama 3.1 8B 4-bit if you're targeting top-of-the-line phones.
• Real-time camera feature: MobileNetV4 for classification, EfficientDet-Lite0 for the easiest object-detection SDK, YOLO11 only if AGPL is acceptable.
• Offline transcription: Whisper large-v3-turbo on recent flagships, smaller Whisper builds below that.

Where to find models and the runtimes that load them

Models are scattered across vendor sites and the Hugging Face hub. The places worth knowing:

• Hugging Face: the biggest model hub. Most modern model cards link to their official mobile builds and benchmarks. Filter searches with gguf, litert, executorch, or mlx to find mobile-ready ones.
• Google AI Edge / LiteRT: official home for Gemma and Gemma 3n mobile builds.
• Meta ExecuTorch examples: the official Llama-on-phone path, with phone-class measurements.
• Qualcomm AI Hub: Snapdragon-optimised builds. Strong signal if your fleet is Android-heavy.
• Apple Core ML Models: Apple's curated gallery. Smaller, but tested for the Neural Engine.
• Argmax WhisperKit: Swift package for Whisper on iPhone, with public benchmarks.
• Ollama and llama.cpp: desktop runtimes. Not for shipping on phones, but the fastest way to try a model on your laptop before committing to the mobile port.

If you've never run a local LLM at all, the cheapest first step is to install Ollama on your laptop, pull a 4-bit Gemma or Qwen build, and ask it to do your task. That answers the quality question without any mobile engineering. If the laptop answer is "no, we need cloud anyway," you've saved yourself weeks. If it's "yes, this is good enough," the mobile port becomes a runtime integration problem, and the runtime is largely chosen for you by the model you picked.

FAQ

What does "local LLM" actually mean?

A local LLM is a language model, like the ones behind ChatGPT or Claude, that runs entirely on the user's device. The model file is downloaded once and lives in app storage. Generation happens on the phone's chip, so nothing the user types or receives leaves the device. Today's mobile-class models are roughly 1–8 billion parameters. Much smaller than the 100+ billion-parameter frontier models that run in the cloud.

What's the smallest useful local LLM I can ship in 2026?

Gemma 3 1B at INT4 (around 657 MB) is the strongest under-1-GB choice. It scores 80.2 on IFEval and 62.8 on GSM8K per Google's model card. Qwen2.5-0.5B at INT8 is smaller still (~521 MB) and Apache 2.0 licensed, which makes it the easiest closed-source option.

What is "quantization" and why does INT4 keep coming up?

Quantization is the trick that makes mobile LLMs work. Every number inside the model normally takes 32 bits to store. Researchers figured out that for running the model (not training it) you can squeeze most numbers down to 4 bits with only a small quality drop. INT4 means "4 bits per number," about 8× smaller than the training format. Modern INT4 techniques like SpinQuant preserve roughly 92% of the original model's behaviour, and QAT+LoRA can do better than that.

Can I just use Apple Intelligence or Gemini Nano instead of bundling my own model?

Sometimes, yes. Apple's Foundation Models framework, introduced at WWDC 2025 and shipping in iOS 26, and Google's AICore / Gemini Nano on supported Android devices give you OS-level access to a local LLM without bundling weights yourself. The catch is device coverage. Apple Intelligence needs an iPhone 15 Pro or newer. AICore launched on the Pixel 8 Pro and Galaxy S24 and similar high-end devices, then broadened to more of the Pixel line. Below that, you're back to bundling your own.

Will running a local LLM kill my user's battery?

It'll warm the phone and draw power, yes. But the real product risk is thermal throttling, not the absolute energy cost. A model generating 30 tokens/second when the phone is cool can drop to 10 once the chip is warm, and that's the slowdown users actually feel. Run on the NPU or Apple Neural Engine, keep sessions short, and don't reload the model on every screen.

Do I really need user consent to download a model file?

Not as a hard EU AI Act requirement for the model file itself. The Act regulates the AI system, not the storage. But the ePrivacy Directive covers storage of information on user devices, app stores expect transparent prompts for large downloads, and a clear "Download local AI model? ~2 GB, Wi-Fi recommended" dialog handles the legal corner case and a thousand support tickets in one go.

Is the EU AI Act going to break my mobile app?

Almost certainly not, if your feature is a general-purpose chat assistant. Most consumer apps land in limited risk and only need transparency under Article 50: a "this is AI" UI affordance and a label on generated content. High risk obligations only apply to the uses listed in Annex III of the Act: biometric identification, employment decisions, access to essential services.

Can I fine-tune one of these models for my app?

Yes, and on the smaller models it's surprisingly cheap. Techniques like LoRA and QLoRA fine-tune a 1B–3B model in hours on a single consumer GPU. The runtime catch is that you either re-quantize after fine-tuning, or ship a small adapter file at runtime alongside the base model. Both LiteRT and ExecuTorch support adapter-style deployment, though the path is less polished than vanilla inference.