What Building Our Own Offline AI App Taught Us About On-Device AI for Clients

Before we build an AI system for a client, we usually build a version of it for ourselves first. Not as a demo, but as a real product we actually use, with real edge cases and real failures. Our offline AI chat app is one of those: a fully on-device mobile assistant, no cloud, no API calls, built with React Native and Expo.

Here's what broke, what we learned, and why it changes how we scope similar work for clients.

The problem that doesn't show up in a demo

Getting a small LLM to generate a plausible response on a phone is the easy part. Most tutorials stop there. What they skip is what happens after: memory pressure, context loss mid-conversation, and models that behave completely differently on an Android low-end device versus an iPhone.

We ran into this directly. On Android, our default inference settings (n_ctx=1024, threads=4, batch=512) work fine for smaller models like Phi-4 Mini Light. Load a larger model (Gemma 4 or full Phi-4 Mini) with the same settings and you get out-of-memory crashes on mid-range devices. We ended up capping context and batch size specifically for larger models (n_ctx capped to 512, n_batch to 128, mlock disabled) just to keep the app stable. That's not a setting you'd find documented anywhere. It came from crashing the app on our own phones enough times to know where the ceiling was. The inverse problem showed up too: our smallest reasoning model got caught by that same "large model" cap by mistake and slowed to a 20-second prefill on every message, until we excluded it explicitly.

Context loss is a real failure mode, not an edge case

Partway through building, we discovered that the llama.cpp context can become invalid mid-conversation: the completion call just throws "Context not found." The first version of the app had no handling for this at all; a conversation would just die.

We built a recovery path: catch the error, release the broken context, reload the model from disk, and ask the user to resend their last message. It's a small piece of engineering, but it's the difference between an app that occasionally, inexplicably stops responding and one that degrades gracefully. This is the kind of failure mode you only find by using the product for weeks, not by testing it for an afternoon.

Speech is its own set of problems

Speech-to-text runs on Whisper.rn, specifically the smallest "tiny" Whisper variant, English-only. We picked it deliberately over the full-size Whisper models: on a phone, the accuracy gain from a bigger Whisper model isn't worth the extra load time and memory for a chat assistant's mic input.

Text-to-speech is the real trade-off in this app. Both voice options run through Sherpa-ONNX, but Piper is small, fast, and bundled in (28MB), while Kokoro is a heavier download (310MB) that gives you 11 higher-quality studio voices instead of one. Neither is a strict upgrade: Piper wins on footprint and startup speed, Kokoro wins on voice variety and naturalness. The right one depends on the device and how much the user cares about voice quality versus app size, a decision we'd rather let the user make than guess ourselves.

What this means for client work

The gap between "a model that answers questions" and "a product people can rely on" is almost entirely made up of the problems above: memory management, failure recovery, engine selection, and platform-specific tuning. None of it shows up in a slide deck. All of it shows up in production.

When a client comes to us wanting an on-device AI agent, a voice assistant, or an offline-capable AI feature, we're not starting from theory. We've already hit the OOM crash, already built the context-recovery path, already had to weigh a voice engine's size against its quality for a real product instead of a hypothetical one. That's the actual value of building your own products before selling the same capability to someone else: fewer surprises in someone else's production environment.

If you're scoping an on-device or offline-capable AI system and want to work with a team that's already run into the problems you haven't hit yet, we're at Riverborn. Happy to talk through what you're building.