RAG Docs Assistant

Documentation Q&A that crawls docs or ingests PDF/DOCX/HTML and answers from a local LLM. No external API, nothing leaves the machine.

Private · write-up

External API calls: 0
Datastores to operate: 1

The problem

The obvious way to build documentation Q&A sends your internal documents to a third-party API. For a lot of documents that is simply not allowed, and the question of whether it is allowed usually arrives after the tool has already been adopted. Building it fully local removes the question.

Architecture

Ingests either a crawled docs site or uploaded PDF/DOCX/HTML, so the same index serves published docs and internal ones.
Token-aware chunking rather than fixed character splits, so a chunk boundary does not land mid-sentence and strand the context a retrieved passage needs.
Semantic search over pgvector, with PostgreSQL holding both the documents and the embeddings — one datastore to run and back up.
Answers are generated by a local model through Ollama, grounded in retrieved chunks. There is no external API call anywhere in the path.

What I chose, and what I rejected

Two choices carried this one, and both are the kind that look like preferences until you hit the case that decides them.

Keeping embeddings in Postgres

The default advice for retrieval is to reach for a dedicated vector database. It is genuinely the right answer at a certain scale — purpose-built ANN indexes, horizontal sharding, and tuning knobs Postgres does not expose.

It is the wrong answer for a documentation corpus, and the reason is not performance. It is that a dedicated vector store makes the embedding a separate system of record from the document. Now you have two datastores to run, two to back up, two to keep consistent, and a class of bug where a document exists but its vector does not, or the vector survives a document that was deleted. Every one of those is a distributed-systems problem you did not have before, taken on to solve a search problem you did not yet have.

pgvector removes the class entirely. The chunk row and its embedding are the same row, written in the same transaction. If the document goes, the vector goes with it, because it is a column.

The payoff that made the decision obvious came later, and it is the thing worth asking me about: metadata filtering happens in the same query as the similarity search. “Find the closest passages, but only from the current version of the docs, and only from pages the caller may read” is one SQL statement with a WHERE clause. In a split architecture that becomes either a pre-filter you push into the vector store’s limited metadata language, or a post-filter that fetches k results and throws most of them away — and when it throws away too many, you re-query with a larger k and hope. Postgres does this with a predicate and an index.

The honest limit: this holds while the corpus fits an exact or IVFFlat scan comfortably. At a scale where recall/latency tuning becomes the dominant problem, a dedicated store starts winning and I would move. For documentation — tens of thousands of chunks, not tens of millions — that point is a long way off, and the operational simplicity is worth more the whole way there.

Token-aware chunking

Fixed-size character splitting is one line of code and it is the single biggest quality bug in naive RAG implementations.

The failure mode is specific. Splitting on characters cuts a code block in half, a table across its header, a numbered list between the step and its explanation. The resulting chunk still retrieves well, because it is full of the right keywords. It is simply useless as context: the model receives half a function signature, or a table body with no header telling it what the columns mean, and answers confidently from a fragment.

That is the worst possible failure shape — high retrieval score, low answer quality — because it is invisible in retrieval metrics. Your recall looks fine. Your answers are wrong.

Chunking on token counts with awareness of document structure means chunk boundaries land where a human would put them, and a retrieved passage is answerable from. It also makes the context budget real rather than approximate: token counts are what the model actually consumes, so a chunk sized in tokens has a known cost and a known fit, while a chunk sized in characters has neither.

Why fully local was the point, not a limitation

The obvious way to build documentation Q&A sends your documents to a third-party API. For a lot of documents that is simply not permitted — and the question of whether it is permitted usually arrives after the tool has been adopted, which is the worst time for it to arrive.

Running embedding and generation through a local model via Ollama removes the question rather than answering it. There is no data-processing agreement to negotiate, no retention policy to read, no region to worry about, and no per-token cost that scales with how useful the tool becomes. The tool is deployable in exactly the situations where documentation Q&A is most valuable and least permitted: internal runbooks, contracts, anything under NDA.

The trade is real and worth stating plainly. A local model is weaker than a frontier model, so the generation step is the quality ceiling here. What that buys is that retrieval quality becomes the lever you can actually pull — and for grounded Q&A over a known corpus, retrieval is where most of the quality lives anyway. A better retrieval pipeline in front of a smaller model beats a worse one in front of a larger model more often than the framing suggests.