How I Built an AI Invoice Extractor with GPT-4o Vision and Nuxt 3

Anyone who has worked on document processing knows the pain of traditional OCR. You set up a coordinate-based parser or write dozens of regular expressions for vendor names, tax IDs, and totals. It works smoothly until a vendor changes their invoice template by 20 pixels or sends a tilted phone scan. Suddenly, your parser extracts the invoice date as the balance due.

To solve this, I built a full-stack AI Invoice Extractor using Nuxt 3, Express, and GPT-4o Vision. Instead of rigid bounding boxes, it treats invoices visually—just like a human accountant would—and pairs vision extraction with a real-time human feedback loop.

Here's how it's built and what I learned along the way.

The pipeline: from scan to structured data

The extraction flow runs in five distinct stages:

1. Ingestion & Multi-Upload — The Nuxt 3 frontend accepts PDF, PNG, JPG, JPEG, and WebP invoices with instant client-side validation.
2. High-Res Rasterization — For PDF files, the backend uses pdf-poppler to render the primary invoice page into a high-resolution PNG, preserving typography, table lines, and stamp overlays.
3. Multimodal Extraction — The image is converted to a base64 payload and sent to GPT-4o Vision (detail: 'high', temperature: 0.1) with a strict schema covering vendor info, dates, payment status, balance due, and line items.
4. Confidence & Anomaly Scoring — The model assesses document clarity, calculating a confidenceScore (0–100%) and flagging any uncertainFields.
5. MongoDB Persistence — Extracted records and verification metadata are saved directly to MongoDB.

Catching visual nuances: the "PAID" stamp problem

Standard text OCR frequently fails on financial context:

• An invoice with a total of $1,500 might have a diagonal red "PAID" watermark across the center. A plain text parser marks it as an open debt.
• A bill might state $2,000 total, with a small note at the bottom: "Deposit: $500. Balance Due: $1,500".

By sending high-resolution images directly to GPT-4o with explicit instructions to inspect stamps, watermarks, and footer balance amounts, the system reliably classifies payment status (paid, unpaid, partially_paid, pending) and calculates the actual balance due.

The Human Feedback Loop

In financial workflows, an undetected mistake is far worse than an explicit error. When image quality is poor or critical fields are smudged, the system doesn't guess silently:

• Uncertainty Alert — If confidence drops below 80% or key fields are missing, the UI raises an amber alert: "AI Confidence is Low — Human Feedback Required", tagging the uncertain fields.
• Inline Editing — The user expands the invoice card to inspect or correct vendor names, amounts, dates, and line items directly in the form.
• One-Click Verification — Clicking "Save Corrections & Verify" updates MongoDB with the human-verified values and sets isHumanVerified: true, switching the badge to a clean green verified state.

What building this taught us

1. Vision beats coordinate OCR for varied templates. Invoices are inherently graphical. Table grids, column spacing, and visual badges provide critical context that text-only scrapers lose.
2. Build for uncertainty from day one. Requiring the AI to estimate its own confidence and flagging uncertain fields prevents bad data from slipping into downstream ledgers.
3. Server-side rasterization is worth the effort. Rendering PDFs to high-res images before model inference gives the AI full visual fidelity without losing fine print or tabular hierarchy.