On-Device RAG Architecture: How Nongor Does Private Document Search

Most "chat with your documents" tools have a hidden cost: your files get uploaded to someone else's servers. For contracts, medical records, case files, or internal strategy docs, that's a dealbreaker. Nongor was built to solve both problems at once, with high retrieval accuracy and no document data leaving the machine.

The Problem With Cloud Document Search

Retrieval-augmented generation is now the default way to answer questions over a private corpus. You split documents into chunks, embed them, store the vectors, retrieve the closest ones to a question, and hand them to a language model to write an answer. The pattern works well, and most hosted products implement some version of it.

The catch is where those steps run. In a typical hosted RAG product, your documents get uploaded, chunked, and embedded on someone else's infrastructure. The vectors live in a database you don't control, and the raw text usually sits alongside them so it can be returned as context. For a marketing team indexing blog posts, that's fine. For a law firm indexing discovery material, a clinic indexing patient records, or a company indexing unreleased financials, it's a compliance problem before it's a technical one.

The usual workarounds are unsatisfying. Redacting documents before upload destroys the detail that makes search useful. Signing a data processing agreement moves the liability without moving the data. Self-hosting a cloud product often still ships embeddings or telemetry outward.

Nongor takes the other route: keep the whole retrieval pipeline on the machine that holds the documents, and treat any network call as something that has to justify itself.

Architecture at a Glance

The pipeline runs in six stages. The first five stay on your device.

1. Chunk and embed. Files get parsed, split into overlapping chunks, and embedded by a local model.
2. Dense and sparse retrieval. A vector search and a keyword search both run against the local index.
3. Reciprocal rank fusion. The two ranked lists get merged into a single candidate pool.
4. Cross-encoder reranking. A reranker model scores each candidate against the actual question and reorders them.
5. Citation validation. Every citation in the draft answer gets checked against the chunks retrieval actually returned.
6. Answer generation. The top chunks go to a language model. Cloud by default, or local through LM Studio.

Step 6 is the one exception to the on-device rule: generating the final answer currently calls a cloud LLM (OpenAI) by default. Installing LM Studio and pointing Nongor at a local model closes that last gap, at the cost of response speed.

Why Everything Runs on Your Device

Retrieval, embedding, and reranking all run locally. Zero bytes of your document data leave the device during those steps. It's the constraint that shaped every other design decision in the system.

Keeping the pipeline local rules out a lot of convenient architecture. You can't reach for a hosted vector database, a managed embedding endpoint, or a hosted reranking API, which are the three services most RAG stacks lean on. Every one of those would mean shipping either the document text or a vector derived from it off the machine.

Embedding vectors are not anonymised text

It's tempting to treat embeddings as safe to transmit because they're arrays of floats rather than readable prose. That intuition doesn't hold. Embedding inversion research has repeatedly shown that a meaningful amount of the source text can be reconstructed from its vector, especially for short chunks. If your threat model says the text can't leave, the vectors can't leave either.

Local models are now good enough

Running embedding and reranking locally used to mean accepting a large accuracy drop. That gap has narrowed. Modern open embedding models and cross-encoder rerankers in the small size classes run comfortably on a laptop and land close enough to hosted alternatives that the privacy guarantee is worth the difference for most document sets.

Ingestion: Chunking and Embedding

Retrieval quality is decided at ingestion time, before any question gets asked. If a chunk splits a table header away from its rows, or cuts a clause in half, no amount of clever ranking downstream will recover the answer.

Why chunk size is a real tradeoff

Small chunks embed cleanly. A short passage about one topic produces a vector that sits close to questions about that topic, so retrieval precision goes up. The cost is context: a small chunk may not carry enough surrounding text for the model to write a complete answer, and a fact that spans two chunks can get split.

Large chunks preserve context but blur the embedding. A page covering four topics produces a vector that's an average of all four, so it ranks mediocre for every one of them. Overlapping windows help by making sure a fact near a boundary appears whole in at least one chunk, at the cost of a larger index.

Structure survives the split

Chunking on document structure beats chunking on a fixed character count. Splitting at headings, sections, and paragraph boundaries keeps semantically coherent units together, and carrying the heading path into each chunk gives the retriever extra signal about where the text came from. A chunk that knows it lives under "Section 4.2 — Termination" is easier to retrieve than the same text floating free.

Hybrid Retrieval: Dense Plus Sparse

Search by meaning alone misses exact terms: invoice numbers, product codes, error messages, statute references. Search by keyword alone misses paraphrased questions. So Nongor runs both.

Dense retrieval handles meaning

The dense retriever embeds the question with the same model used at ingestion and finds the chunks whose vectors sit closest. Because it works on meaning rather than surface form, it answers questions phrased in words that never appear in the source. A user asking "can we get out of this early?" can match a clause titled "Termination for convenience" even with zero shared vocabulary.

The weakness shows up with rare, precise strings. Identifiers like INV-2024-8871 or ERR_CONN_RESET carry almost no semantic content, so the embedding model has little to work with and the right chunk can rank poorly.

Sparse retrieval handles exact terms

The sparse retriever does classical term matching, scoring chunks by how often the query's terms appear relative to how rare those terms are across the corpus. Rare terms count for more, which is exactly the behaviour you want for identifiers, proper nouns, and error codes. It has no idea what words mean, so paraphrased questions can slip past it entirely.

The two failure modes barely overlap

Dense and sparse retrieval fail in different situations, which is what makes running both worthwhile. A question that defeats one method usually lands well with the other, so the union of their results has better coverage than either list alone. That's the case for hybrid search in one sentence: two retrievers with uncorrelated blind spots.

Reciprocal Rank Fusion

Running two retrievers leaves you with two ranked lists and a merging problem. Their scores aren't comparable. A cosine similarity of 0.82 and a keyword relevance score of 14.3 live on different scales with different distributions, so you can't add, average, or threshold them without inventing a normalisation scheme that needs retuning every time a model or corpus changes.

Reciprocal rank fusion sidesteps the scale problem by throwing the scores away and keeping only the positions. Each chunk gets a contribution based on where it ranked in each list, with a constant that damps the influence of the very top positions, and the contributions get summed. A chunk that placed third in both lists can outrank a chunk that placed first in one and nowhere in the other.

Two properties make it a good fit here. It needs no training data and no per-corpus tuning, so it behaves consistently when someone swaps the embedding model. And it rewards agreement between retrievers, which is a reasonable proxy for relevance when you have no labels to learn from.

Cross-Encoder Reranking

After fusion, a cross-encoder reranks the candidates so the most relevant chunks surface first, right before they get passed along to generate an answer.

Bi-encoders vs. cross-encoders

The embedding model used for retrieval is a bi-encoder. It encodes the question and each chunk separately, then compares the resulting vectors. That separation is what makes retrieval fast: chunks get embedded once at ingestion, and answering a query means one embedding plus a nearest-neighbour lookup. The cost is that the model never sees the question and the chunk together, so it can't reason about how they relate.

A cross-encoder does see them together. It takes the question and one chunk as a single input and produces a relevance score with full attention across both. That's a much better judgement, and much more expensive, because it runs once per candidate instead of once per query.

Retrieve wide, rerank narrow

The pattern that makes this affordable is to let the cheap stage cast a wide net and the expensive stage make the final call. Fusion produces a candidate pool big enough that the right chunk is almost certainly somewhere in it, then the cross-encoder scores that pool and promotes the genuinely relevant ones. You pay cross-encoder cost on a bounded number of candidates rather than the whole index.

This stage is also where borderline retrieval mistakes get caught. A chunk that ranked well because it shares vocabulary with the question, without actually answering it, tends to score poorly once a model reads both together.

Citation Validation

An answer with a citation is only trustworthy if the citation is real. So Nongor checks every citation against the exact chunk it was retrieved from. If a citation doesn't map to a real, retrieved source, it doesn't get shown. Invented references and plausible-sounding fake page numbers never make it through.

Grounding is a rendering decision, not a prompt

Most systems try to prevent fabricated citations by asking the model nicely, with instructions like "only cite the provided sources." Instructions reduce the rate; they don't make it zero, and the failures that get through are the convincing ones. A citation that points at a real document with a wrong page number reads exactly like a correct one.

Treating validation as a rendering step changes the failure mode. Every reference in the draft answer gets resolved against the set of chunks retrieval actually returned. A reference that resolves gets displayed with a link back to its source text. A reference that doesn't resolve gets dropped. The reader never has to evaluate whether a citation is real, because an unreal one can't be displayed.

Citations you can open

Because every displayed citation maps to a specific retrieved chunk, it can link to that chunk's text. That turns verification into one click rather than a search through the original document, which matters when the answer is going into something that gets reviewed.

The Evaluation Harness

An evaluation harness runs on every update. Recall targets check whether retrieval still finds the right chunks. An LLM-as-judge scores whether answers stay faithful to the source material. If a change drops either score below the bar, we catch it before it ships, not after a user hits it.

Why RAG systems need this more than most software

A RAG pipeline has no compiler and no type system for quality. Change the chunk size, swap the embedding model, adjust how many candidates go to the reranker, and everything still runs. Nothing throws. The system just gets quietly worse at some class of question you won't notice until someone complains. Without measurement, tuning a RAG stack is guesswork with confident narration.

Two things worth measuring separately

Retrieval and generation fail for different reasons, so they get scored separately. Retrieval is measured against a labelled set: for a given question, did the chunk containing the answer make it into the results? That isolates the pipeline up to reranking and answers the question "was the information even available to the model?"

Generation is measured for faithfulness: given the chunks that were retrieved, does the answer actually follow from them, without adding claims the sources don't support? A separate model reads the answer alongside its sources and scores the alignment. Scoring these separately matters because the fixes are different. Bad retrieval is a chunking or ranking problem. Unfaithful generation with good retrieval is a prompting or model problem.

The harness is a gate, not a report

Running evals and reading the numbers afterwards catches regressions eventually. Running them as a gate catches regressions before they ship. Each run is compared against the current baseline, and a change that drops recall or faithfulness below the threshold gets blocked rather than logged.

Cloud LLM vs. Fully Local LLM

Retrieval, embedding, and reranking run on-device today. The final answer-generation step currently uses a cloud LLM, OpenAI. That means the retrieved chunks, not your full documents, get sent to OpenAI to write the answer.

The distinction is worth being precise about. Your corpus is never uploaded. Your index stays local. What crosses the network in the default configuration is the handful of chunks selected as context for one specific question, plus the question itself. For many teams that's an acceptable exposure, and it buys the speed and quality of a frontier model. For others, any document text crossing the boundary is the thing they're trying to avoid.

Running generation locally with LM Studio

If you want the entire pipeline to run without any data touching a cloud LLM, install LM Studio on your device and point Nongor at a model running there instead of OpenAI. That keeps generation local too, so nothing leaves your machine at any stage.

The setup is short:

1. Install LM Studio on the machine running Nongor.
2. Download a model that fits your available memory.
3. Start LM Studio's local server, which exposes an OpenAI-compatible endpoint.
4. Point Nongor at that endpoint instead of the OpenAI API.

Because LM Studio speaks the OpenAI API format, this is a configuration change rather than a code change. Retrieval and reranking are already local, so this closes the last gap.

The tradeoff: a local LLM runs a little slower than a cloud LLM, since it's using your own hardware instead of a data center GPU. That's worth knowing before you pick a setup. For document sets that can't leave the building, the extra wait is usually a fair trade.

Comparing the two setups

Tradeoffs We Accepted

Every architectural choice here costs something, and it's more useful to name those costs than to pretend the design is free.

• Ingestion is slower. Embedding a large corpus on a laptop takes longer than firing it at a hosted endpoint with a fleet of accelerators behind it. Indexing is a one-time cost per document, so this lands mostly on first-run experience.
• Query latency is higher than single-retriever RAG. Running two retrievers, fusing them, and then reranking with a cross-encoder is more work than one vector lookup. The accuracy is worth it, and the reranker stage is the part to tune if latency becomes a problem.
• Local embedding models trail the largest hosted ones. The gap is smaller than it was and keeps closing, but it exists. For corpora where the last few points of retrieval accuracy matter more than locality, Nongor can use OpenAI embeddings instead. That's a deliberate switch, not a default.
• Hardware sets the ceiling. Everything on-device means everything constrained by the device. A workstation with a discrete GPU has a very different experience from a thin laptop, particularly once generation is local too.

What This Gets You

• Zero bytes leave the device during chunking, embedding, retrieval, fusion, and reranking.
• Hybrid dense and sparse retrieval, so exact identifiers and paraphrased questions both work.
• Every citation shown maps to a real retrieved source, with a link back to that text.
• Recall and faithfulness get checked automatically on every update, before a change ships.
• An option to go fully local, with no network calls at any stage, via LM Studio.

FAQ

Does Nongor send my documents to the cloud?

No. Chunking, embedding, dense and sparse retrieval, fusion, reranking, and citation validation all run on your device. The only step that can reach the network is final answer generation, and you can keep that local too by running a model in LM Studio.

What is hybrid retrieval and why does it matter?

Hybrid retrieval runs a dense (vector) search and a sparse (keyword) search over the same index and merges the results. Dense search handles paraphrasing and meaning. Sparse search handles exact strings like invoice numbers, part codes, and error messages. Running both catches cases either method alone would miss.

What is reciprocal rank fusion?

Reciprocal rank fusion merges two or more ranked lists into one by scoring each document from its position in each list rather than from raw similarity scores. Because it uses rank position, it avoids the problem of comparing vector similarity against keyword relevance, which are on different scales.

How does Nongor prevent hallucinated citations?

Every citation in a generated answer is checked against the set of chunks that retrieval actually returned. If a citation doesn't resolve to a real retrieved chunk, it isn't displayed. That makes a fabricated reference a rendering failure rather than something the reader has to catch.

How do I run Nongor fully locally?

Install LM Studio, download a model, start its local server, and point Nongor at that endpoint instead of OpenAI. Retrieval and reranking are already local, so this closes the last gap. Expect slower responses than a cloud model, since generation runs on your own hardware.

Is a local LLM slower than a cloud LLM?

Yes. A local model runs on your own CPU or GPU instead of a data center accelerator, so answers take longer to generate. The exact difference depends on your hardware and the model size you choose.

Riverborn builds AI systems for teams handling documents that can't leave the building. Book a discovery call.