Overview
On April 23, 2023, we started HeyLuna, five months after ChatGPT launched and weeks after GPT-4 arrived. That was the beginning of the AI era as most people would come to know it. Before November 2022, large language models were a research curiosity. By spring 2023, the world was waking up, but almost nobody had figured out what came after chat.
The idea was straightforward but ambitious: build a personal AI assistant that could handle real-life problems managing emails, scheduling calendar events, sending messages, and staying on top of daily tasks through natural voice and text conversation, not dashboards and menus.
At the time, this category did not exist at all. There were no AI agents, no “connect your Gmail” products, no agent frameworks to copy. People were still asking “what can ChatGPT do?”, not “what can AI do on my behalf?” We saw that second question coming, bet on it before the market did, and started building.
Three years later, personal AI assistants are everywhere from email copilots to calendar agents to voice-first companions. That validation doesn’t diminish what we did. It confirms our instincts were right. The hard part was never the idea. It was building in a world where the tools, the patterns, the users, and the category itself did not yet exist.
The Problem
Modern knowledge workers drown in fragmented tools:
- Inboxes that never stay at zero
- Calendars that require constant manual upkeep
- Context switching between email, chat, search, and scheduling apps
- No single interface that understands intent and takes action
Existing productivity software was built for the pre-AI world: click-heavy, form-based, and disconnected. In 2023, even the idea of an AI taking actionnot just generating text was novel. Users didn’t need another chat window. They needed someone who could understand “email Sarah about tomorrow’s meeting and block an hour on my calendar” and just do it. Nobody was offering that yet.
Our Vision
HeyLuna is not a chatbot bolted onto a dashboard. It is a 3D virtual personal assistant with a persistent personality, long-term memory, and the ability to act across the services people already use.
Users talk to Luna naturally, by voice or text, and Luna responds in character while executing real tasks in the background: reading and sending email, creating calendar events, searching the web, checking weather, finding places, and posting to Slack.
The experience is designed to feel human: short, conversational replies, a customizable avatar with behavioral traits, and memory that carries across sessions.
Business Challenges
When we started HeyLuna in April 2023, the term “AI agent” was not in mainstream use. There was no playbook, no category, and no market ready to buy what we were building. We were inventing a product type before the world knew it wanted one.
- No Market Category to Stand On – Personal AI assistants that act on email, calendar, and messaging did not exist not as products, not as a concept. Every pitch and landing page had to create the category from scratch while also selling the product inside it.
- User Education Before User Acquisition – Most people had never used an LLM, or had tried one for writing. The idea of AI reading their inbox or sending Slack messages on their behalf required a mental leap. We were competing against disbelief, not competitors.
- Trust Is the Product – HeyLuna touches email, calendar, work chat, preferences, and conversation history. One wrong action a mis-sent email, a deleted event, a privacy concern could end the relationship permanently.
- Monetization Without a Pricing Playbook – No standard existed for pricing a personal AI assistant. We designed freemium credits, premium tiers, and integration-gated features while the entire AI SaaS pricing model was still being invented.
- Controlled Rollout vs. Growth Pressure – We launched with a waitlist and authorization flow onboarding users into an AI that accesses personal accounts requires capacity planning, support readiness, and quality control that scaling too fast would break.
- First Mover Cost, Late Mover Competition – Being early gave us a head start on architecture and integrations, but also meant years of R&D cost, infrastructure investment, and waiting while the market caught up to the vision.
- Relationship Product in a Utility Market – HeyLuna is a companion a 3D avatar with personality and memory that also manages email. Users evaluate utilities on features; they evaluate companions on trust and habit. Neither had a proven playbook.
Technical Challenges
There was no playbook on the engineering side either. We were discovering the architecture as we built it.
- AI + Personal Data Is a Different Problem – Connecting an LLM to Gmail or Google Calendar required OAuth flows, tool-calling architecture, guardrails for destructive actions, and context management with no production reference in early 2023.
- Real-Time Voice + AI + Actions – Speech transcribed in real time, processed through streaming LLM responses, converted back to speech, and synchronized with a 3D avatar all over WebSockets with session persistence and interruption handling.
- Long-Term Memory at Scale – Persistent AI memory was not a solved problem in 2023. We implemented vector-based memory with Pinecone, combined with MongoDB session history and dynamic system prompts incorporating user profile and past context.
- Multi-Provider Integration Complexity – Users don't live in one ecosystem. Gmail, Google Calendar, Outlook, and Slack each with OAuth consent screens, scope reviews, and production access requirements managed through Nango.
- Personality, Not Just Utility – Each user configures an avatar with behavioral traits and communication style. The system prompt dynamically assembles Luna's personality while executing precise tool calls behind the scenes.
- Production Infrastructure Before PMF – A personal AI assistant handling OAuth tokens, streaming voice, and destructive actions on user accounts cannot run on prototypes. We deployed on Cloud Run with multi-region Azure OpenAI failover while the product was still finding its audience.
- OAuth & Platform Approval – Connecting to Gmail, Calendar, Outlook, and Slack means app verification processes and ongoing compliance months of friction that pure SaaS products never face.
- Unpredictable AI Costs at Scale – Every voice conversation triggers Deepgram, multiple LLM calls, TTS, Pinecone lookups, and third-party API calls. Unit economics shift with every conversation length and complexity.
- No Reference Architecture for Agentic Assistants – OpenAI's function calling API was two months away. LangChain was experimental. We invented tool schemas, prompt engineering, error recovery, confirmation flows, and the entire agent loop from scratch.
All over WebSockets (Socket.IO) with session persistence, credit management, and interruption handling.
What We Built
Tool-Calling Layer
We built a tool-calling layer with 14+ defined functions from getEmails and sendEmail to addEventToCalendar, getEvents, Outlook mail, Slack messaging, web search, news, weather, and place lookup each gated by the user’s connected integrations and subscription tier. Destructive actions require clear intent; context management keeps user preferences isolated per account.
Integrations
| Provider | Capabilities |
|---|---|
| Gmail (read, send, delete), Google Calendar (create, read, delete events) | |
| Microsoft | Outlook mail (read, send) |
| Slack | Channel and direct messaging |
| Default | Web search, news, weather, place lookup |
OAuth and connection lifecycle are managed through Nango, with per-user integration state stored in MongoDB. Premium users unlock integrations based on what they’ve connected; free-tier users get core utility functions.
Core Capabilities
- Voice & text conversation with streaming AI responses
- Email management read, search, send, delete (Gmail + Outlook)
- Calendar management create, query, and delete events
- Slack messaging to channels and users
- Web utilities search, news, weather, place lookup
- Customizable 3D avatar with personality configuration
- Long-term memory via vector search over conversation history
- Subscription tiers with integration-gated premium features
- Session history with persistent conversation storage
Technology Stack
- Runtime: Node.js 18, Express
- Real-time: Socket.IO over WebSockets
- AI / LLM: OpenAI, Azure OpenAI (multi-region failover)
- Speech: Deepgram (STT), Azure Cognitive Services (TTS)
- Auth & data: Firebase Admin, MongoDB
- Memory: Pinecone vector search over conversation history
- Integrations: Nango (OAuth lifecycle), Gmail, Calendar, Outlook, Slack
- Search: Typesense, Serper
- Deployment: Google Cloud Run, Cloud Build, Fly.io, Secret Manager
Results & Validation
For the first two years, we built largely alone in a space that did not exist. Then the market caught up. Since 2024, the entire industry has moved toward exactly what we envisioned: AI agents that connect to personal accounts and take action on behalf of users.
The vocabulary caught up too “AI agents,” “tool use,” and “function calling” are standard terms today. In April 2023, they were not. That shift validates our original thesis. HeyLuna was early. It was hard. And it was right.
Key Takeaways
- Timing matters, but conviction matters more. We built before agents, before function calling, before the category existed because we understood the problem deeply, not because the market told us to.
- Business and technical challenges are the same problem. Trust, pricing, and market education are not separate from OAuth security, voice latency, and tool-calling reliability.
- Personal AI is an integration problem first. The LLM is the interface; the hard work is OAuth, token refresh, function schemas, and safe action execution across third-party APIs.
- Voice + action + personality is multiplicative. Each layer alone is manageable. Together they require careful pipeline design especially when none of the underlying infrastructure is mature yet.
- Memory transforms a chatbot into an assistant. Vector retrieval over past conversations makes Luna feel like someone who knows you. We built this before the industry decided memory was essential.
- Being early is a business cost, not a mistake. First-mover advantage in knowledge and architecture comes with years of expense in education, infrastructure, and uncertainty.
- Building at the frontier has compounding returns. The research, architecture, and integration patterns developed for HeyLuna are directly applicable to the agentic AI products the industry is building today.