Offline AI Chat : Fully On-Device, Privacy-First AI

Overview

Most AI chat apps assume a connection to the internet: type a message, it goes to a server, a model runs there, a response comes back. That’s true for ChatGPT, Gemini, and nearly every AI feature bolted onto another product. We wanted to know what it would take to skip that entirely — an AI chat app where the model runs on the phone, with no server involved at any point after setup.

The result is a React Native (Expo) mobile app with four pieces working together: chat inference on @pocketpalai/llama.rn (React Native bindings for llama.cpp), voice input through either Whisper.rn or Sherpa-ONNX, voice output through either Piper or Kokoro, and MobX-managed state persisted to AsyncStorage. All four pieces had to work without ever assuming a network connection was available, which affected nearly every part of the build. The only place the app touches the network at all is the very first run, downloading the chosen model — after that one download, the app never needs a connection again, verified directly by putting the phone in airplane mode and using it normally.

Business Challenges

• Privacy Users Won't Compromise On – A growing set of users want AI help but won't accept their conversations, voice recordings, or personal context leaving their phone. For that audience, a cloud-dependent chat app isn't a worse option — it's a non-starter regardless of how good the model is.
• AI That Works With Zero or Unreliable Connectivity – Flights, rural coverage, data caps, and corporate networks that block AI endpoints are all real, common conditions. The bar isn't “degrades gracefully offline” — it's fully usable with no connection at all, indefinitely.
• One App, Wildly Different Hardware – The same binary has to run on a three-year-old budget Android phone and a current flagship. With no server to elastically absorb the difference, the app itself has to make the trade-offs, and make them honestly rather than crashing silently.

Technical Challenges

• Running a Multi-Gigabyte Model With No Server to Fall Back On – A phone has no elastic memory pool and no swap to rely on. Generic, desktop-sized context and batch settings caused Gemma 4 (5.41GB) to crash reliably on mid-range Android with nothing but an out-of-memory error in the logs.
• Silent Context Invalidation Mid-Conversation – The underlying llama.cpp context can become invalid mid-conversation on mobile. The first version of the app had no handling for this: a conversation would simply stop working with no explanation.
• Native Modules That Expo Go Can't Touch – llama.cpp bindings, Whisper.rn, and Sherpa-ONNX are compiled native code. Expo Go can't load any of it, which forces a custom development build workflow and native patches (via patch-package) from day one, not as an afterthought.
• Two Real-Time Pipelines With No Objectively Right Answer – Speech-to-text and text-to-speech each have two viable on-device engines with opposite trade-offs — accuracy versus latency, footprint versus language coverage. Committing to one engine per pipeline means guessing wrong for a meaningful share of users.

Our Approach

Our design philosophy was “assume nothing about the network, and budget real engineering time for the 80% that isn’t the model.”Getting a model to generate a plausible response on a phone is the easy part; every tutorial gets you that far. The harder, unglamorous work is memory management, failure recovery, and platform differences that only surface once real people are using the app, not demoing it. Where there wasn’t a clear “best” option — speech-to-text, text-to-speech, even which LLM to run — we shipped both viable engines and let the device and the user decide, rather than guessing wrong for a chunk of them.

Discipline & Contributions

System Architecture

The app has no backend of its own. Four tabs — Chat, Models, Settings, Talk — read and write MobX stores that are persisted to AsyncStorage, which in turn call straight into native inference libraries running inside the app process.

The one honest exception is the very first run.We didn’t hide this — onboarding walks a new user through picking an LLM and a TTS voice, shows real download progress via a background downloader so the transfer survives the app going to the background, and only completes onboarding once the model is actually on disk and loaded into memory. On every launch after that, the app checks the filesystem only — no request is made for a model that’s already downloaded.

Engineering Deep-Dives

1. Zero-Server Chat: How a Response Actually Gets Generated

When a user sends a message, ChatSessionStore.addMessage() records it and creates an empty assistant placeholder. ModelStore.generateCompletion() builds a prompt from the system prompt, the last N messages, and the new message, then calls context.completion() with a token callback. Each token updates the message in the store, and MobX re-renders the UI immediately, producing the streaming effect users expect — without a single network request in the loop.

2. The Out-of-Memory Crash That Forced Real Memory Engineering

A phone isn’t a server: there’s no elastic memory pool, no reliable swap, and background apps are already competing for RAM before the model even loads. We saw this most clearly with Gemma 4, our largest model at 5.41GB — generic settings worked fine on a high-end test device and crashed reliably on a mid-range one, with nothing but “out of memory” in the logs. Three settings decide whether a model fits: context window (n_ctx), batch size (n_batch), and memory locking (use_mlock). None of our final numbers came from documentation — they came from loading a model, watching it crash, lowering a value, and repeating that loop on real devices until it stopped happening.

3. Recovering From a Dead Context Instead of Dying With It

Even with tuned settings, the llama.cpp context can still go invalid mid-conversation on mobile. The failure surfaces as a thrown error — “Context not found” — and the first version of the app had no handling for it: a conversation would simply stop working with no explanation. We built a recovery path instead: catch the error, release the broken context, reload the model fresh from disk, and ask the user to resend their last message.

4. Getting Past Expo Go: Native Builds, Patches, and largeHeap

Expo Go, the usual way to preview an Expo project instantly, only supports the native modules Expo ships by default. llama.cpp bindings, Whisper.rn, and Sherpa-ONNX are all custom compiled native code, so Expo Go can’t load any of it — the app requires a custom development build (npx expo run:android / npx expo run:ios) from day one. Some native dependencies also needed fixes upstream hadn’t merged yet. We rely on patch-package for this — a patch for @dr.pogodin/react-native-fs applies automatically on every yarn install via a postinstall hook. We also explicitly enabled largeHeap: true in app.jsonfor Android, without which the default heap ceiling isn’t enough for a multi-gigabyte model plus the rest of the app running normally.

5. Downloads That Survive the App Being Backgrounded

Model files run from hundreds of megabytes to several gigabytes. A standard fetch tied to the app staying in the foreground isn’t viable — users background the app and the download stalls. We use @kesha-antonov/react-native-background-downloaderspecifically so downloads continue when the app isn’t active, and so the app can detect and resume an in-progress download if it’s reopened mid-transfer. TTS models with extra files (tokens.txt, voices.bin, espeak-ng-data.zip) download sequentially and are extracted automatically once complete.

6. Two Engines Each, By Design, Not By Accident

Text chat is the easy half of an offline AI assistant. Voice is where the real trade-offs show up, because unlike picking one LLM, there isn’t a clear winner for either speech-to-text or text-to-speech. Whisper.rn is more accurate on accents and background noise but heavier and slower to initialize; Sherpa-ONNX is lighter with a better live-streaming feel but needs more tuning to match Whisper’s accuracy. Piper (Amy Low,63MB) is small, fast, bundled, and English-only; Kokoro (344MB) covers nine languages at the cost of size and a slower cold start. We didn’t pick a winner — we shipped both pairs and defaulted sensibly: Whisper and Piper as the safer default, Sherpa-ONNX and Kokoro as the upgrade path.

Configuration & Performance Gates

Technology Stack

• Framework & Language: React Native + Expo (~53), TypeScript, Expo Router (file-based navigation)
• State Management: MobX + mobx-persist-store, backed by @react-native-async-storage/async-storage
• LLM Inference: @pocketpalai/llama.rn (llama.cpp bindings) — TinyLlama, Phi-3 Mini, Phi-4 Mini (Light & full), Gemma 4 (E2B & E4B), Gemma 2B
• Speech-to-Text: whisper.rn (Whisper) and react-native-sherpa-onnx (streaming ONNX STT)
• Text-to-Speech: react-native-sherpa-onnx running Piper VITS (Amy Low voice) and Kokoro v1.1 multilingual models
• Downloads & Storage: @kesha-antonov/react-native-background-downloader, expo-file-system, @dr.pogodin/react-native-fs
• Build Tooling: patch-package (native module patches applied via postinstall), EAS Build for production Android/iOS

Results & Outcomes

Key Takeaways

• Offline isn’t a feature flag, it’s an architecture decision. It has to hold at every layer — chat, voice in, voice out, and state — not just the model call.
• The model is the easy 20%.Memory tuning, context recovery, and platform-specific limits are the other 80%, and they’re what decide whether the feature survives contact with a real user’s phone.
• When there’s no objectively best engine, ship both and default sensibly. Letting the device and the user choose beats guessing wrong for a meaningful share of them.