Nongor : Local-First Knowledge Base

Overview

Nongor ingests documents and web pages, indexes them on the machine it runs on, and answers questions in plain English — with a verifiable citation behind every claim. The product hinges on two things most RAG demos skip: answers you can trust and data that never leaves the box. A single embed, a single vector search, and a single LLM call will produce a plausible-sounding paragraph — but with no way to know whether it is grounded, and only after shipping your documents to someone else’s cloud.

So rather than trusting one similarity search, Nongor stacks independent retrieval signals — dense vectors, sparse BM25, and a cross-encoder reranker — and then grounds every generated claim in a citation that is checked against the retrieved text. Everything except the final generation call runs locally with zero network egress, and even that last hop is designed to swap to a fully offline model with no code change. This study focuses on the two parts that took the most engineering: the retrieval & answer pipeline and the local-first data boundary.

Business Challenges

• Data Cannot Leave the Building – Legal, healthcare, and defense teams sit on exactly the documents a knowledge base is most useful for — and are precisely the teams who cannot ship those documents to a third-party cloud API. Most RAG SaaS is a non-starter before the first demo.
• An Answer You Cannot Trust Is a Liability – A confident, ungrounded answer over a contract clause or a patient record is worse than no answer. Teams need to see exactly which source sentence backs every claim before they act on it.
• Knowledge Trapped in Documents – Answers already exist inside long PDFs, wikis, and specs — but finding them means a human re-reading dozens of pages, so the same questions get re-researched again and again.

Technical Challenges

• Retrieval Is the Ceiling – If the right passage never makes it into the prompt, no model can answer correctly. Pure vector search quietly misses exact terms — part numbers, error codes, proper nouns — that a user will search verbatim.
• Hallucinated & Unsupported Claims – LLMs confidently synthesise facts that are not in the source, and will even fabricate a citation identifier that looks valid. Without a check, a made-up reference reaches the user looking exactly like a real one.
• Context Lost at Chunk Boundaries – Naive fixed-size chunking severs a table from its header and strands a pronoun from the entity it refers to, so the embedded chunk no longer means what the document meant.
• Everything Must Run on One Machine – Embedding and reranking models have to load and run reliably on commodity hardware — including CPU and Apple Silicon — with no cloud GPU to fall back on.

Our Approach

The design philosophy was “verifiable answers, on your own hardware.” Rather than treating accuracy as a model property, Nongor treats it as a system property: independent retrieval signals compound, a strict citation contract makes the model show its work, and a validation step catches any claim whose reference does not resolve. Local-first is a hard constraint, not a marketing line — retrieval, embedding, reranking, and storage run with no egress, and a startup guard refuses to boot in local mode if the LLM endpoint points at the public internet.

Discipline & Contributions

The Ingestion Pipeline

Retrieval quality is decided long before a question is ever asked — at ingestion. Nongor turns every source into clean markdown, chunks it without destroying its structure, and enriches each chunk so it stays findable even from queries that share none of its wording.

• Format-aware parsing. PDFs, Office files, and HTML go through Docling, which preserves tables and headings as markdown; web pages use trafilatura with a headless Playwright fallback for JavaScript-rendered content.
• Idempotent by hash. Every document is fingerprinted with a SHA-256 of its normalised markdown; re-ingesting an unchanged file is a no-op, so sweeps and re-uploads never duplicate the corpus.
• Structure-preserving chunking. A section-aware chunker walks the markdown heading tree and windows text to ~400 tokens with 50-token overlap — but treats tables and fenced code blocks as atomic, never splitting a cell or a function in half.
• Contextual-prefix indexing.A small LLM writes a 50–100 token standalone context for each chunk (“this paragraph continues the discussion of X from section 2”) which is prepended before embedding. Generated in parallel at concurrency 8, it degrades gracefully to the plain chunk if a call fails — better recall, never a hard dependency.

The Retrieval & Answer Pipeline

This is the centrepiece. A question does not go straight to a vector search — it passes through an intent router, two independent retrieval engines, a cross-encoder, and finally a generation step whose every citation is verified before it reaches the user.

• Intent routing. A fast heuristic plus a small-LLM classifier sends greetings and general questions straight to a no-retrieval path, so the corpus is only searched when searching it actually helps — and general chat stays instant.
• Hybrid retrieval. Dense vector search (BGE-M3, top-50) and sparse BM25 full-text search (top-50) run in parallel and merge via Reciprocal Rank Fusion (k=60). Dense catches paraphrase; BM25 catches the exact tokens — part numbers, error codes, proper nouns — that embeddings blur.
• Cross-encoder rerank. The fused candidates are rescored by a BGE-reranker-v2-m3cross-encoder down to the top-8 that actually enter the prompt — a second, stronger relevance signal than first-stage similarity, computed on the exact query–chunk pair.
• Grounded generation & citation validation.The prompt pins the model to cite each claim with its source chunk’s bare UUID. After generation, a regex extracts every cited id and checks it against the retrieved set: valid ids become clickable citations; fabricated ones are flagged as invalid_citation_ids. A hallucinated reference cannot reach the user disguised as a real one.

Local-First by Construction

For the industries Nongor targets — legal, healthcare, defense — where the data goes is as important as whether the answer is right. So the boundary is drawn in the architecture, not in a privacy policy: only one call in the entire system is capable of leaving the machine, and it is trivial to close.

The practical upshot is simple: anyone unwilling to put sensitive documents in a public or third-party cloud database can run Nongor on their own hardware and keep the corpus entirely to themselves. The data is indexed and searched locally, so no one else — not a SaaS vendor, not Riverborn, not another tenant — can view or access it.

• What stays on the box. Parsing, chunking, embedding (BGE-M3), reranking, the LanceDB store, and citation validation all run locally — a corpus can be indexed and queried with the network cable pulled.
• The single chokepoint. Only call_llm() talks to a model provider, so pointing Nongor at a fully local runtime (Ollama, vLLM, LM Studio) is a config change, not a refactor — the provider is an OpenAI-compatible endpoint behind one env flag.
• The egress guard. With LLM_PROVIDER=local, the server refuses to start unless the LLM base URL is a loopback or private address — so a copy-pasted cloud key cannot silently ship documents off the machine.
• Swappable at runtime. Embedding and reranking can flip between the local models and cloud services from the settings panel without a restart, for teams that want to trade a little privacy for speed on non-sensitive corpora.

Measured, Not Asserted

Accuracy claims are only as good as the harness behind them. Nongor ships with its own report card: a hand-labeled question set and an evaluation runner that scores retrieval, faithfulness, and citation validity, reports bootstrap 95% confidence intervals, and fails CI on a ≥3% relative regression against the last run.

The Rest of the System

The retrieval pipeline and the data boundary are the centrepiece, but they sit inside a complete application — an async FastAPI backend, a streaming three-panel workspace, and a provider abstraction that keeps the whole thing portable across LLM vendors.

• Ingestion & workspace.Documents arrive by direct upload, URL, or CLI; a background task streams live progress (“chunking… embedding… saving”) while the three-tab UI — Chat, Upload, Sources — surfaces retrieved passages next to every answer.
• Provider-agnostic core. Any OpenAI-compatible endpoint works — DeepSeek, OpenAI, Ollama, vLLM, LM Studio — selected by a single env flag, with all model traffic funnelled through one function so a vendor swap touches exactly one file.
• Async & streaming ops. FastAPI serves answers over Server-Sent Events with typed status, delta, citations, and timings events; embedding and reranking run in worker threads so the event loop never blocks. The whole stack ships as a Docker Compose file.

Technology Stack

• Frontend & UI: Nuxt 3, Vue, Tailwind CSS, streaming chat with live source panel
• Backend Orchestrator: Python 3.11, FastAPI, Uvicorn (async), Pydantic config, structured logging
• Retrieval & Models: BGE-M3 embeddings, BGE-reranker-v2-m3 cross-encoder, hybrid dense + BM25, RRF fusion
• Data & Store: LanceDB (vector + full-text + metadata), tiktoken chunking, SHA-256 idempotency
• LLM & Ops: Any OpenAI-compatible provider (DeepSeek / OpenAI / Ollama / vLLM), Docling parsing, Docker Compose

Results & Outcomes

Takeaways

• Retrieval is the ceiling — invest there. Hybrid dense + sparse retrieval fused with RRF, then a cross-encoder rerank, beats a single vector search. Independent signals compound; the LLM can only be as good as what you put in the prompt.
• Ground every claim, then verify it.A strict citation contract plus post-hoc validation turns “trust me” into “check the source,” and makes a hallucinated citation a detectable event rather than a silent one.
• Local-first is an architecture, not a toggle. A single LLM chokepoint and a boot-time egress guard mean the whole system collapses to fully offline with no code change — privacy you can prove, not promise.
• LLMs as components in deterministic plumbing. The model writes prose and contextual prefixes; retrieval, fusion, reranking, citation-checking, and the eval gates all live in plain, testable, measured code.