Vocalo : Ultra-Low Latency Voice AI

Overview

Vocalo enables users to learn English through real-time spoken conversations with Nuri, a friendly and charismatic AI speaking partner. However, in spoken learning applications, the margin for error is razor-thin. Pauses longer than 1.5 seconds feel unnatural and make the session feel like a series of disjointed voicemails rather than a live discussion.

Historically, Vocalo was built around a traditional WebSocket-based stateful orchestration. The client recorded audio blobs, sent them in chunks to a Node.js backend, which compiled the files, hit OpenAI's Whisper API, sent the output to GPT models, waited for completion, and forwarded sentences to Azure and ElevenLabs TTS APIs. While functional, the multi-hop latency averaged 3.0 seconds, and handling user interruptions was incredibly brittle.

To address these core latency bottlenecks, Riverborn re-engineered Vocalo's speech pipeline. We migrated the backend to a unified WebRTC streaming architecture using Pipecat and Daily.co. By replacing the multi-hop API architecture with a single declarative streaming pipeline, we cut response latency to 1.4 seconds and implemented native, sub-100ms interruption handling that instantly silences the bot when the user starts speaking.

Business Challenges

• Real-Time Interaction or Nothing – English learners use Vocalo to build conversational confidence. In a face-to-face conversation, pauses longer than 1.5 seconds feel unnatural and discouraging. Traditional REST architectures cause frustrating gaps.
• Clean Listening In Crowded Environments – Users practice on the go—on buses, in coffee shops, or in busy classrooms. Background noise, echo from speakers, and overlapping ambient voices disrupt the AI's listening accuracy and lead to nonsensical responses.
• Dynamic Translation Mid-Conversation – When beginners get stuck, they need a visual safety net: the ability to see a Spanish, Bengali, or Arabic translation of Nuri's response immediately, without pausing the auditory flow of the session.

Technical Challenges

• Conquering the Multi-Hop Audio Latency Wall – Moving audio from Frontend Mic → STT API → Backend Orchestrator → LLM generation → TTS compilation → Frontend Player is a bottleneck. In a naive system, these steps are sequential, pushing latencies past 4 seconds.
• Natural Interruption Handling – Humans speak dynamically. If the AI is speaking a long sentence and the user asks a quick question, the AI must halt instantly. Safely flushing state, cancelling active TTS audio frames, and resetting context queues without crashing audio streams is incredibly difficult.
• Acoustic Feedback Loops – Without headphones, the sound coming out of the device's speaker leaks back into the microphone. This causes the AI to hear its own voice, transcribe it, and reply to itself in an infinite feedback loop.
• Zero-Latency Cold Starts – Waiting for the backend room to boot, tokens to initialize, and the LLM to generate its first greeting creates a 3-5s initial freeze. The user experience must feel instant from the moment they hit 'Start Practice'.

Our Approach

Our design philosophy focused on “conversational fluidness and hardware isolation.” We treated voice conversation as a single, continuous stream rather than a collection of static files. We replaced HTTP/WebSocket gateways with WebRTC peer connections. To streamline code maintenance, we transitioned from ad-hoc synchronization queues in Node.js to a declarative, event-driven pipeline orchestrator written in Python using the Pipecat framework.

Discipline & Contributions

The Latency Bottleneck: Old vs. New

Comparing the two architectures illustrates the performance difference. The old architecture was plagued by sequential execution delays, whereas the new pipeline processes audio in a single stream.

Why WebRTC & Pipecat Slashed Latency

• Pipelined execution instead of serialization: In the old WebSocket setup, the system had to wait for the user to completely stop speaking, write the file, and send it to Whisper. In the new model, DeepgramSTTService runs a live transcription connection. Words are recognized in real-time as they are spoken, which primes the LLM context beforehand.
• Eliminating HTTP Connection Overhead: Instead of executing a sequence of independent REST API calls, the server uses a persistent WebRTC connection established with Daily.co, streaming audio frames over UDP using RTP/SRTP.
• Native Interruption Processing: When the user speaks while the bot is outputting audio, the pipeline detects incoming user audio frames. Pipecat instantly flushes the downstream audio output buffers and cancels the remaining tokens in the LLM queue, allowing the bot to stop speaking and listen immediately.

Engineering Deep-Dives

1. Acoustic Interference Cancellation (AIC SDK)

When users practice speaking without headphones, speaker output feeds back into the microphone. Software-based echo-cancellation is critical to prevent the agent from transcribing its own voice.

We integrated the hardware-accelerated AIC Filter directly within the Daily WebRTC transport pipeline using the quail-vf-2.1-l-16khz model. The filter isolates and cancels outgoing bot audio before the STT processor reads the incoming feed.

2. Zero-Latency Startup Greetings

To avoid the initial 2-3 second latency of boot-up room authorization and LLM first-token generation, we short-circuit the initialization flow. When the Daily transport fires the on_first_participant_joined event, the backend bypasses the LLM and queues a pre-compiled text string directly to the TTS generator. Nuri says hello immediately while the LLM context is loaded and prepared in the background.

3. Mid-Stream Asynchronous Translation

For beginner English learners, seeing translations of Nuri's speech is key. However, running a translation model in the main speaking thread would block audio streaming. We designed an asynchronous background pipeline to handle translations out-of-band.

• Incremental Compilation: As the LLM generates response tokens, they are compiled on-the-fly.
• Background Translation Task: The moment the LLM fires the LLMFullResponseEndFrame, the completed sentence is passed to Google Translate via an asynchronous task in the background.
• UI Event Dispatch: Once translated, the text is pushed to the client using a Daily.co appMessage channel. The front-end receives and renders the translation side-by-side with the transcript without delaying audio playback.

4. Structured Linguistic Evaluation & Feedback Engine

Vocalo doesn't just let users speak; it actively helps them improve. The moment a speaking practice session is closed, the system triggers an asynchronous feedback collection workflow via the /api/evaluate/session/ endpoint. Instead of providing general summaries, we leverage structured model schemas to extract detailed analytics:

• Qualitative Evaluation (evaluateConversation): Using OpenAI function calling, the model reviews the complete session context. It generates an overall second-person performance summary and builds concrete suggestions: 8-10 specific things the user did well, 8-10 grammar mistakes, and 8-10 actionable improvement steps.
• Quantitative Analysis (analyzeSpeechTranscription): The backend runs a secondary structured query to compile numeric scores from 1 to 100 on five key dimensions: Pronunciation, Vocabulary, Grammar complexity, Fluency, and Minimal usage of filler words.
• Asynchronous Translation: The compiled evaluations are persisted in MongoDB. Beginners can toggle translations dynamically. On request, Google Translate API is queried in the background to translate the qualitative feedback arrays without blocking the app's responsiveness.
• IELTS & CEFR Band Score Mapping: To ground scores in academic standards, the UI runs a mapping utility (evaluateLanguageSkills) that converts the raw performance sub-scores (0-100) into equivalent **IELTS 9-band scores** (e.g., Band 6.5, 7.5) and CEFR proficiency levels (e.g., \"Expert\", \"Very Good\", \"Competent\") alongside targeted feedback paragraphs.

5. The Adaptive AI-Driven Curriculum & Test Generator

Traditional language-learning apps use static tests that do not adapt to individual speech habits. In Vocalo, feedback is directly connected to a personalized learning loop. The backend processes the evaluation scores to dynamically generate custom vocabulary and grammar practice quizzes targeting the user's specific weaknesses:

• Threshold-Based Triggering: After a session, the orchestrator evaluates the user across four categories: fillerWord, fluency, grammar, and vocabulary. If the sub-score for any category drops **below 90**, it triggers a custom quiz generation task.
• Dynamic Quiz Generation: The system compiles the list of errors (such as detected filler words or grammar mistakes) and queries OpenAI in the background to generate multiple-choice quiz questions based on the user's actual mistakes, saving the results in a MongoDB tests collection as `in-progress`.
• Interactive Practice Loop: The Nuxt 3 frontend retrieves these generated tests dynamically by summaryId, rendering a custom Vue template (e.g. test-grammar.vue or test-vocabulary.vue) for targeted review. Once the quiz is completed, the user's updated results are saved to close the practice loop.

6. Multi-Locale Global SEO & Nuxt i18n Strategy

To support users worldwide, Vocalo's landing and content pages are internationalized across 8 locales (English, Bengali, French, Spanish, German, Russian, Korean, and Japanese) using Nuxt 3 and @nuxtjs/i18n:

• Dynamic Browser Locale Detection: A custom language detector (localeDetector.ts) processes request queries, cookie stores, and browser headers (accept-language) to route incoming visitors automatically without jarring manual redirects.
• SEO-Friendly Metadata & Indexing: The site configures subfolder routing, translating document titles, headers, and description tags in all target languages for index crawling on localized search engines.

Real-Time Performance Gates

To ensure conversational fluidness, the voice pipeline is configured with strict thresholds.

Technology Stack

• WebRTC Transport & Rooms: Daily.co REST API, Daily Meeting Tokens, @daily-co/daily-js client SDK
• Pipeline Orchestration: Pipecat AI (Python), PipelineTask, PipelineRunner, FrameProcessor pipeline API
• Audio Processing & Filters: AIC SDK (Acoustic Interference Cancellation), quail-vf-2.1-l-16khz model
• Model Integrations: Deepgram STT (Nova-2 live streaming), Deepgram TTS (Aura voice streaming), OpenAI GPT-4o
• Translation Layer: Google Cloud Translation API (async REST translation triggered on LLMFullResponseEndFrame)
• Frontend & Web Backend: Nuxt 3 (Vue), Tailwind CSS, FastAPI (Uvicorn), MongoDB, Docker & Cerebrium Serverless Runtime

Results & Outcomes

Key Takeaways

• Conversational UI is about streaming, not files. Moving away from buffers and files to WebRTC UDP streams is essential for human-like response latencies.
• Interruption is a core system requirement. A voice buddy that ignores the user when they speak feels robotic. Interruption must be handled natively at the transport level, not as a custom frontend patch.
• Decouple speech from secondary capabilities. Features like translations or database writes should never block the audio pipeline. Run them out-of-band using background workers and events.