InvoiceAgent : AI Document Intelligence Engine

Overview

InvoiceAgent takes a messy scanned or emailed invoice, reads it accurately, turns it into trustworthy structured data, and then lets users ask questions about that data in plain English. The whole product hinges on one hard thing: transcription accuracy. Invoices are visually inconsistent — PDFs, photos, scans, every layout imaginable — and a wrong total is worse than no answer.

OCR on invoices is not “solved.” A single OCR pass plus a single LLM call gets you a demo, not a product: silent numeric errors (1,250.00 read as 1.250,00), hallucinated dates, flattened line-item tables, and no confidence signal to decide what to trust. So rather than trusting one model, Riverborn built the platform around redundancy and cross-validation — two independent OCR engines, two LLM extraction passes, and a comparison step. When the passes agree, the invoice auto-completes at full confidence; when they disagree, a human decides. This study focuses on the two parts that took the most engineering: the OCR + LLM extraction pipeline and the natural-language search layer.

Business Challenges

• Fragmented Data Ingestion – Accounts teams had to manually download, open, and key in invoices from multiple email inboxes and portals, causing delay and double entry.
• Financial Accuracy Risk – Even tiny errors in values, dates, or vendor identities lead to incorrect bookkeeping, duplicate payments, or incorrect tax submissions.
• Lack of Spending Visibility – Finance teams could not easily aggregate or query invoice metadata, meaning forgotten SaaS subscriptions or duplicate contracts went unnoticed.

Technical Challenges

• Silent Numeric Translation Errors – Standard OCR models frequently confuse thousand/decimal separators (e.g. 1.250,00 vs 1,250.00), creating incorrect numeric data silently.
• Structural Layout Disruption – Flattening visually complex document tables into text breaks cell bounds, making line-item reconciliation extremely difficult.
• LLM Hallucination Guardrails – Enforcing strict schema structures on natural language queries without generating invalid parameters or incorrect categories.

Our Approach

Riverborn conducted detailed discovery sessions with accounts teams to map out invoice processing bottlenecks. The resulting design philosophy was “guaranteed transaction integrity.” Rather than trusting a single AI prediction, the backend is built to cross-verify outputs from distinct OCR structures, flagging mismatches immediately to prevent silent accounting errors.

Discipline & Contributions

The Extraction Pipeline

The key move is that every job runs two independent transcription paths and compares them. Each OCR output is run through its own LLM extraction, in parallel, before the results are reconciled. The two engines matter because their errors are uncorrelated— when both independently land on the same total, currency, and payment status, that agreement is a far stronger accuracy signal than one model’s self-reported confidence.

• Two independent OCR engines. Path A runs mistral-ocr-latest, returning clean per-page markdown that preserves table structure; Path B runs a separate markitdownmicroservice on Cloud Run — a genuinely different engine with different failure modes.
• Strict-schema LLM extraction with provider fallback. Both outputs are fed through one defensive extraction contract that pins the model to a single JSON shape — numbers as numbers, omit unknown dates rather than guess, infer currency from symbols, and model VAT/tax/discount as explicit line items so totals stay auditable. The call uses Claude and fails over to GPT-4(with Gemini available) so one provider outage doesn’t stall extraction.
• Cross-validation & confidence gating. A compareSummarize step checks the three fields that matter most for trust — total, payment status, currency. Agreement pins confidence to 100 and writes the invoice; disagreement parks the job in review_needed with both summaries stored. Duplicate invoices are caught here too, by matching invoice_id within the org.

Human-in-the-Loop Review

Full automation is the wrong goal for financial data — a confidently-wrong total is the most dangerous output the system can produce. So the pipeline is built to know when it isn’t sure and hand off to a person. Because both candidate extractions are pre-computed, review is a bounded pick-one comparison — seconds, not data entry from scratch — and every resolution doubles as a labeled example of where the models diverged.

The job lifecycle

Processing is asynchronous and modeled as an invoice_jobsrow walking a fixed lifecycle. The valid states aren’t a convention — they’re a database CHECK constraint, so a buggy worker physically cannot persist an invalid status.

Natural-Language Search

Once invoices are structured, users want to askthings: “how much did we spend on AWS last quarter?”, “show unpaid invoices over $1,000.” Keyword search can’t aggregate, filter, and do math over structured records, so NLQ is built on pandas-ai, which translates the English question into pandas code and runs it against the org’s dataframe.

The hard part wasn’t generating the query — it was that pandas-ai returns different result types depending on the question: a raw number, a table, a text blob, or a generated chart image. None of those is a user-facing answer on its own, so each type is routed through Claude to produce a natural caption tailored to its shape.

The Rest of the System

The OCR/LLM and NLQ pipelines are the centerpiece, but they sit inside a full multi-tenant SaaS backend — one mobile-grade ingestion backbone, a Node/Express API that houses every engine, and metered billing that gates each feature.

• Ingestion & email. Invoices arrive via direct upload, Gmail, and Outlook (OAuth via Google and Azure/MSAL, brokered by Nango). A scheduled job sweeps connected inboxes for attachments; OAuth tokens are AES-encrypted before they ever hit the database.
• SaaS spend “tech-scan.” The same backbone scans a connected inbox for vendor signals, using a Claude Haiku classifier to label each email (paid / trial / renewal / overdue) — the same pattern as the invoice pipeline: an LLM as a confidence-gated classifier inside deterministic plumbing.
• Billing, multi-tenancy & ops. Stripe-backed metered credits gate every feature; Firebase verifies ID tokens on every protected route; every table carries org_id through a parameterized CRUD layer that centralizes SQL-injection safety. Deployed to Cloud Run via Cloud Build with secrets in GCP Secret Manager.

Technology Stack

• Frontend & UI: React, Next.js, Tailwind CSS, Interactive Chart components
• Backend Orchestrator: Node.js 18, Express, PostgreSQL (Render), Google Cloud Storage
• AI & Extraction: Claude 3 Sonnet/Haiku, GPT-4 fallback, Mistral OCR, Custom markitdown Microservice
• Database & Logic: PostgreSQL, pgvector, CHECK state machine constraints, pandas-ai
• Integrations & OAuth: Google OAuth 2.0 (Gmail API), Outlook API, AES-encrypted tokens, Stripe Billing

Results & Outcomes

Takeaways

• Transcription accuracy is the product. We invested in redundancy — two OCR engines, two LLM passes, cross-validation — rather than trusting one model. Uncorrelated agreement beats self-reported confidence.
• Give the LLM structure, not freedom. A strict JSON schema, markdown-preserving OCR, and explicit modeling of VAT/discounts removed entire classes of numeric error.
• Always have an escape hatch. Provider fallback (Claude → GPT-4), a review_neededhuman gate, and a deterministic guided-search alongside pandas-ai mean no single component is a hard dependency.
• LLMs as components in deterministic plumbing. Both the extraction validator and the NLQ caption layer use the model for fuzzy judgment while keeping orchestration, gating, and persistence in plain, testable code.

What the client says