Conversation Platform : Live Chat and an AI Bot in One Widget

Overview

Riverborn sells a customer-conversation platform to small businesses. A visitor lands on a customer’s website, opens a chat bubble, and talks to either a human agent or an AI trained on that business’s own content. The business manages everything from a single dashboard.

Two things made this harder than a typical CRUD SaaS. The product runs inside someone else’s website — third-party JavaScript on domains we don’t control. And the marketing site shipped before the product did. “Under 8 kB widget,” “install in 60 seconds,” data in GCP Mumbai (asia-south1), “no raw IPs stored,” “unlimited conversations” stopped being copy and became engineering acceptance criteria.

The bet that paid off best was designing the schema for three conversation modes — live_agent, rag_bot, hybrid — before building any of them. When the AI product was pulled forward, it needed Qdrant, embeddings, and an LLM layer. It did not need a schema migration, a new widget protocol, a new auth model, or a second inbox.

Business Challenges

• Two products, one team, one codebase – Livechat was supposed to ship first, AI later. Market pull inverted that mid-build. Both had to exist without doubling the surface area.
• Pricing a product whose unit cost is variable – Livechat has near-zero marginal cost, so “unlimited conversations” was safe. Every AI reply costs embeddings, vector search, and LLM tokens. The same pricing page had to carry both.
• The product runs inside someone else’s website – The widget is third-party JavaScript on domains we don’t control. It must never break the host, leak styles, or slow the load — and the public snippet is frozen the moment a customer pastes it.
• Marketing shipped before the product – A live landing page promised “under 8 kB,” “install in 60 seconds,” data hosted in GCP Mumbai (asia-south1), “no raw IPs,” “unlimited conversations,” “free plan, no credit card.” Copy became acceptance criteria.

Technical Challenges

• An 8 kB gzip widget budget – That number eliminated the Firebase JS SDK, any UI framework, and shared types before a line was written. The shipped widget is one ~890-line TypeScript file.
• Realtime without polling, without WebSockets – 1.5s polling worked and was unshippable. The requirement is server→client push only, authenticated with a Firebase ID token — which EventSource cannot send.
• Anonymous identity that doesn’t survive – Visitors are Firebase anonymous UIDs. A cleared refresh token mints a new UID and every read against the existing thread returns 403 — the worst possible returning-visitor UX.
• Firestore → MongoDB without a client rewrite – Nested document paths became top-level collections. Tenant isolation moved from database rules into route middleware. The HTTP shape had to stay still.
• A RAG stack whose embedding model is a hard coupling – Ingestion and query must use the same model. Change one without re-embedding the corpus and retrieval degrades into nonsense rather than failing loudly.

Our Approach

One workspace / project / conversation / message schema serving all three modes from day one, with mode-specific config nulled out rather than modelled separately. One dashboard, with sections unlocked by defaultProduct and enabledProducts[]. Agent takeover is a field write: the dashboard sets handledBy = 'agent', posts a system message, and the bot stops replying. No new tables, no migration.

The embed snippet carries a project ID and nothing else. Everything the widget needs — colour, position, bubble style, greeting, mode, whether there’s a bot — is fetched at runtime from GET /v1/widget/:projectId/config. Changing a customer’s widget never requires them to touch their site again. The widget build copies widget.js into the dashboard’s public/, so both ship in a single Cloudflare Pages deploy.

Discipline & Contributions

The 8 kB Widget

The hard limit is 8 kB gzipped for third-party JavaScript running on a customer’s page. That number eliminated most of the obvious toolkit before a line was written.

Visitor authentication is hand-rolled against the Firebase REST API — signUp for a new anonymous identity, refreshToken to restore one — with a refresh token in localStorage. That’s roughly 40 lines replacing ~40 kB of SDK. Isolation is a closed Shadow DOM, not an iframe: complete style encapsulation in both directions, no second document, bubble animations that can sit on the page as a natural element. What it gives up is an origin boundary. A hostile host page could interfere. For a product installed voluntarily by the site owner, that’s an acceptable threat model.

Contrast handling was a small but necessary detail — customers pick arbitrary brand colours, so getContrastColor() computes readable foreground text rather than assuming white on the accent. There is no automated size check. Vite prints the gzip size at build time and a human is expected to notice. Given the budget is a public promise, this belongs in CI.

Replacing Polling with SSE

The first working version polled GET .../messages every 1.5 seconds while the panel was open. Read volume scaled linearly with concurrent open chats. 1.5s worst-case latency on a product whose value proposition is immediacy.

SSE, not WebSockets: the requirement is server → client push only. Visitor messages still go up over ordinary POST. SSE is plain HTTP, survives proxies that mangle WebSocket upgrades, and the parser already existed — the widget’s token-by-token bot streaming had already implemented an SSE-over-fetch reader. EventSource is unusable here because it cannot send an Authorization header; every stream is authenticated with a Firebase ID token, so the client reads the body as a ReadableStream and splits on \n\n manually.

Keepalive is a : pingcomment every 25 seconds so intermediaries don’t reap an idle connection. The API sends X-Accel-Buffering: no as a backstop; SSE dies silently behind any proxy that buffers responses.

Anonymous Identity That Doesn’t Survive

Visitors are Firebase anonymous UIDs. Authorization is conversation.visitorId === token.uid. This breaks in ordinary conditions: the refresh token in localStorage can be cleared or revoked, at which point the widget mints a new UID and every read and write against the existing conversation returns 403. Naive handling produces a blank, permanently broken thread.

Email becomes the durable identity and the UID becomes a rotating credential. The same mechanism delivers cross-device resume for free — start on mobile, continue on desktop, same thread. The greeting bubble is suppressed on resume. Without that, a returning visitor sees “Hey 👋 How can we help?” above a conversation they were already having.

Firestore → MongoDB, Zero Client Changes

Mid-project the datastore moved from Firestore to MongoDB Atlas. Firebase Auth stayed — UIDs remain foreign keys throughout, so the auth surface was untouched. Nested document paths flattened into top-level collections. Chatbot config, a singleton at a fixed path in Firestore, became an embedded chatbot field on the project document.

What the migration gave up: Firestore security rules enforced tenant isolation at the database layer. MongoDB has no equivalent, so all of it moved into route middleware — requireMember, requireAdmin, requireOwner, requireVisitor. Every authenticated route now does a member lookup and verifies the resource chain. It works, but a single route that forgets a filter is a cross-tenant data leak, where previously the database would have refused the read. Message immutability, previously a rule (allow update, delete: if false), is now the absence of PATCH and DELETE routes.

Keeping the Inbox Consistent

The inbox list renders entirely from denormalized fields on the conversation document — lastMessagePreview, lastMessageAt, lastMessageSenderType, unreadByAgent, messageCount. It never reads the messages collection. That’s what makes the list a single indexed query instead of N subqueries.

Denormalization is only safe if it can’t drift, so message insert and conversation update happen in one MongoDB transaction. The conversations change stream then fires with the preview already updated, so the UI never shows a conversation bumped to the top with stale preview text.

The scaling ceiling is honest: each SSE client opens its own change streams — one on messages, one on conversations. N open tabs means up to 2N streams against Atlas. Right call for an MVP (no fan-out layer, no sticky sessions). The path past that is a shared stream per workspace with in-process fan-out, then a dedicated realtime service. There is also an unresolved deploy contradiction: the realtime design assumes Compute Engine (no per-request timeout, no concurrency-slot limit) while the Dockerfile targets Cloud Run, which has both.

The RAG Pipeline

Retrieval is a tool call, not a preamble. Rather than always prepending retrieved chunks, the model decides when to call search_knowledge_base. Chit-chat doesn’t burn a vector search, and the model can issue a refined second query when the first result set is thin.

• Ingestion is fire-and-forget. Adding a knowledge source returns 201 immediately and processes in the background, with a processing → ready | error status lifecycle. Chunking is 400 words with 50-word overlap, embedded 20 chunks at a time to stay inside OpenAI rate limits.
• Re-index cooldowns are a billing control.Free 24h, growth 6h, scale 1h. Re-indexing re-embeds every chunk from scratch. Without a cooldown, a free user clicking “reindex” in a loop is an unbounded OpenAI bill.
• Provider isolation made churn cheap. The git history shows Anthropic → Groq → restructure, twice. Each provider sits behind its own thin lib/ module.

Pricing the Thing That Costs Money

Livechat has near-zero marginal cost per conversation. The AI chatbot does not. Three decisions came out of that:

• Meter AI replies, not conversations. The counter is usage/{YYYY-MM}.aiMessages. Counting conversations is trivially gameable — open one thread, send two hundred messages. Metering the thing that actually costs money aligns price with COGS.
• Separate Stripe subscriptions per producton one Stripe customer per workspace. A customer can buy livechat, the AI bot, or both, and cancelling one doesn’t touch the other.
• Deferred writes in the creation wizard.The 5-step AI chatbot wizard writes nothing until step 5 (“Launch”). Abandoned sessions leave no orphaned projects, no half-built knowledge bases, and no confusing empty states.

Data Residency as an Architecture Constraint

Data lives in GCP Mumbai (asia-south1), not the EU. That landed on the architecture rather than on a policy page: MongoDB Atlas in Mumbai, M10+ (a replica set is required for transactions and change streams anyway, so the tier was forced from two directions); continuous backup with 12-month retention, matching the “1 year chat history” promise; no raw IP is ever written.

Geolocation is currently satisfied vacuously: it isn’t implemented, so country and cityare stored as empty strings. The promise isn’t violated, but the feature isn’t delivered either. Similarly, the GDPR consent notice has its CSS class and privacy-policy URL computed in the widget, but is never inserted into the DOM. The plumbing exists; the last line is missing.

Architecture at a Glance

Technology Stack

• Widget: Vanilla TypeScript, Vite lib mode (IIFE), closed Shadow DOM, Firebase REST auth, SSE-over-fetch
• Dashboard: React 19, Vite, Tailwind 4, Cloudflare Pages — widget.js copied into public/ so both ship as one artifact
• API: Express on Cloud Run, MongoDB Atlas (GCP Mumbai, asia-south1, M10+ replica set), Firebase Auth, Stripe, Resend, Slack
• AI: Qdrant, OpenAI text-embedding-3-small, Groq / Anthropic LLMs, tool-calling RAG, 400-word chunks with 50-word overlap
• Realtime: MongoDB Change Streams → SSE; per-connection streams; : ping keepalive every 25s; X-Accel-Buffering: no

Results & Outcomes

What We’d Do Differently

• Put the widget size budget in CI.It’s a public promise enforced by a human reading build output. It has already grown 23% since SSE landed.
• Rate-limit before launch, not after. Public endpoints that trigger paid LLM calls need a throttle on day one.
• Treat spec documents as code or delete them. Stale specs are worse than no specs — a new contributor trusts them. Three docs still disagree on the production URL.
• Resolve Cloud Run vs Compute Engine before scaling SSE. The realtime design assumes no request timeout and no concurrency slots; the deploy target has both.

Takeaways

• Schema the modes before you pick a favourite. Designing for live, bot, and hybrid up front meant the AI product reused the entire conversation pipeline when the market pulled it forward.
• Marketing copy is an SLA.“Under 8 kB” and “install in 60 seconds” are not taglines. They are constraints that delete frameworks, SDKs, and entire onboarding write-paths.
• Meter COGS, not vanity units. Conversations are free to count and free to game. AI replies are what OpenAI invoices.
• Decouple the paste-once snippet from behaviour. A project ID and a runtime config fetch means every customer who already installed keeps working when you change the product.
• A compatibility shim can buy you an independent migration. Emitting Firestore timestamps from MongoDB was technical debt with a clear payoff date — the clients never moved.