Offline Voice: Why We Only Ship One Speech-to-Text Engine, and Two Text-to-Speech Voices

Text chat is the easy half of an offline AI assistant. Voice is where the real engineering trade-offs show up, but not evenly. Speech-to-text and text-to-speech turned out to need very different design decisions in our offline AI chat app.

Speech-to-text: one engine, on purpose

We ship a single speech-to-text engine, Whisper.rn, specifically the smallest "tiny" Whisper variant, English-only. That was a deliberate choice, not a limitation we're glossing over. Whisper's larger variants are more accurate, but on a phone, the extra load time and memory of a bigger model isn't worth it for what's essentially mic input for a chat assistant. The tiny model is fast to initialize and transcribes live speech well enough for real conversation.

We did leave an alternative speech-to-text component in the codebase, wired for Sherpa-ONNX, from an earlier direction on the project. It was never finished and isn't used anywhere in the shipping app today. Worth mentioning because it's a good example of a decision every team building this kind of app has to make: build the second option properly, or cut it and ship the one that works.

Text-to-speech: the real trade-off

Text-to-speech is where we actually ship two options, because here the trade-off is real. Both voices run through the same Sherpa-ONNX engine underneath, but they're a genuinely different choice.

Piper (the Amy Low voice) is small, 63MB, fast to synthesize, and bundled with the app rather than downloaded separately. One voice, English only, optimized for footprint and startup speed.

Kokoro is a heavier download at 310MB, and gives you 11 higher-quality studio voices (American and British, male and female) instead of one. It's also English-only in the version we ship, contrary to what you might expect from Kokoro's reputation as a multilingual model upstream, we're running the English-only build. The trade-off is size and cold-start time in exchange for noticeably more natural, varied speech output.

If footprint and speed matter most, Piper is the obvious choice. If voice quality and variety matter more than app size, Kokoro is worth the extra download.

Why the shape of these two decisions is different

Speech-to-text didn't need a real choice because the trade-off space is narrow: for mic input on a phone, a small, fast, accurate-enough English model wins over a more accurate but heavier one almost every time. Text-to-speech needed a real choice because the two options serve genuinely different priorities (footprint versus quality) and neither dominates the other.

The lesson generalizes: don't build optionality into a voice stack just because it feels more complete. Build it where the trade-off is real, and cut it where one option is just obviously right for your use case.

If you're building a voice product and trying to figure out which parts actually need a real engine choice, that's evaluation work we do at Riverborn.