A RAG pipeline is seven small stages in a row: parse, chunk, embed, index, retrieve, rerank, then generate with citations. Most wrong answers come from the first five, before a language model sees anything. This post goes through each stage with a decision table, runs a complete pipeline on a tiny made-up corpus, and ends with a checklist for sorting retrieval misses from generation errors.
If you want the concepts first, read how RAG works. If you want a beginner build over PDFs, there is a walkthrough that runs. This one is the architecture view: what to decide at each stage, and how to see which stage broke.
What does a RAG pipeline look like end to end?
documents -> parse -> chunk -> embed -> index (offline, repeat on change)
question -> embed -> retrieve top-k -> rerank -> prompt with citations -> LLM (online)
Two loops, different failure costs. The offline loop is cheap to rerun but easy to forget: stale indexes are a classic silent failure. The online loop is where latency and token cost live.
I searched for what current guides cover before writing this (2026-10-23). The search summaries I could see were from vendor and practitioner blogs, not research papers, and I did not open every page. They agreed on a few things: hybrid retrieval plus a reranker is the common production setup, chunking is described as the knob that matters most, and you should evaluate retrieval separately from generation. They disagreed on chunk sizes, and one cited a roughly nine-point MRR gain from hybrid plus rerank that the summary itself flagged as a single unverified benchmark. I treat all of those numbers as unverified and measure on my own data below. Sources: Stack AI's RAG best practices, DEV Community: RAG in production in 2026, Interexy's implementation guide.
Stage 1 and 2: parse and chunk
Parsing decides what text exists at all. Scanned pages, tables and headers are the usual losses, and the PDF chatbot post shows a scanned file silently contributing nothing. Log characters per page.
Chunking decides what unit of text can be retrieved. A chunk that mixes two topics retrieves for both and answers neither. A chunk cut mid-sentence loses the fact.
| Decision | Start here | Change it when | Cost of getting it wrong |
|---|---|---|---|
| Boundary | Document structure first (headings, paragraphs), hard character cut only for oversized sections | Source has no structure (transcripts, chat logs) | Facts split across chunks |
| Max size | A few hundred tokens, then test two or three sizes | Answers need long context (legal clauses, procedures) | Too big: diluted match. Too small: no context |
| Overlap | Small (10 to 20 percent) and only on hard cuts | Structure-aware splitting already respects sections | Duplicate chunks eat top-k slots |
| Prefix | Prepend the heading or document title to each chunk | Chunks already say what they are about | Fragments with no subject |
| Runts | Drop or merge chunks under about 40 to 100 characters | Never skip this check | Meaningless fragments rank in the top k |
These are starting points I'd test, not findings. For a measured comparison of fixed, paragraph and heading-aware chunking on real manuals, see chunking strategies compared.
Stage 3 and 4: embeddings and index
An embedding model turns text into a vector so that similar meaning lands nearby. Use the same model for documents and queries, store the model name next to the index, and re-embed everything if you change it.
| Decision | Start here | Change it when |
|---|---|---|
| Model | A small local model for a first build | Your top-3 misses on real questions are about meaning, not wording. Then compare candidates, see embedding models compared |
| Index | A NumPy matrix for up to a few thousand chunks | You need filtering at scale, shared access, or the matrix stops fitting in memory. See FAISS, Chroma, Pinecone and the HNSW and IVF explainer |
| Metadata | Store source file, section, and date on every chunk | Always. You need it for citations and for filtering stale documents |
If you do not want to run a model locally, any OpenAI-compatible embeddings endpoint works with the same code shape. Disclosure: the AICredits link below is an affiliate link; it costs you nothing. AICredits is an Indian gateway with an OpenAI-compatible base URL (https://api.aicredits.in/v1) that takes INR via UPI from Rs 50. I have not run embeddings through it for this post, so check its catalog for an embedding model before you depend on it. OpenAI's own API, Voyage, Cohere and a local model are non-affiliate alternatives.
Stage 5 and 6: retrieve top-k, then rerank
Retrieval returns the k nearest chunks, always k, whether or not any are relevant. A reranker re-scores those candidates with a model that reads the question and each chunk together, which is slower but sharper than comparing two pre-computed vectors.
| Decision | Start here | Change it when |
|---|---|---|
| top-k to retrieve | 10 to 20 candidates if you rerank, 3 to 5 if you do not | Recall at k is below your target on your questions |
| top-n after rerank | 2 to 4 chunks into the prompt | Answers need several sources |
| Reranker | None at first | Gold chunk is usually in the top 10 but rarely in the top 3 |
| Refusal rule | A minimum score tuned on answerable vs unanswerable questions | Always. Without it the system answers from noise |
| Keyword + vector | Vector only | Users search by IDs, codes, names. See the hybrid search measurements |
Stage 7: prompt with citations
Number or label each chunk with its source, tell the model to answer only from them, to cite the label, and to say a fixed phrase when the context lacks the answer. The prompt wording matters and is covered in prompt engineering for RAG pipelines. What matters for debugging: keep the exact prompt that was sent, so you can read it later.
Can I run all of it? A complete pipeline
Everything below ran on 2026-10-23 in a fresh Python virtual environment. Install: pip install fastembed numpy.
What is real and what is stubbed:
- Real: chunking, local embeddings (
BAAI/bge-small-en-v1.5through fastembed), cosine retrieval, and a local cross-encoder reranker (Xenova/ms-marco-MiniLM-L-6-v2through fastembed). - Stubbed: the LLM. I had no API key, so
stub_llmis not a model. It returns the context sentence that shares the most words with the question. It exists only to show the plumbing. - Synthetic: the three documents describe a made-up product. The questions and gold sections are mine.
import re
import numpy as np
from fastembed import TextEmbedding
from fastembed.rerank.cross_encoder import TextCrossEncoder
# SYNTHETIC docs for a made-up product. Markdown with headings.
DOCS = {
"billing.md": """# Billing
## Changing your card
Open Billing, choose Payment method and enter the new card. The next invoice uses it.
## Refunds
Annual plans can be refunded in full within 14 days of purchase. Monthly plans are not refunded, but you keep access until the period ends.
## Cancelling
Choose Cancel plan under Billing. The account becomes read only for 30 days, then files are deleted.""",
"security.md": """# Security
## Two-factor sign-in
Three wrong codes lock sign-in for fifteen minutes. Admins can reset a locked user from the Users page.
## Encryption
Files are encrypted in transit with TLS and at rest with AES-256. Business plan teams can bring their own keys.
## Lost devices
Admins on the Business plan can remotely wipe a device. Other plans can sign the device out from the web.""",
"sync.md": """# Sync
## Ignoring files
Create a .fernignore file in a folder and list patterns, one per line. Matching files are never uploaded.
## Bandwidth
Preferences, then Network, lets you cap upload and download speed in kilobytes per second.
## Conflicts
If two devices edit one file, both versions are kept and one is renamed with the suffix conflicted copy.""",
}
def chunk(doc_name, text, max_chars, overlap):
"""Split on headings first, then hard-split long sections with overlap."""
out = []
for sec in re.split(r"\n(?=## )", text):
sec = sec.strip()
title = sec.splitlines()[0].lstrip("# ").strip()
body = " ".join(sec.splitlines()[1:])
if not body:
continue # heading-only section (the "# Title" line): nothing to index
step = max(1, max_chars - overlap)
for i in range(0, max(len(body), 1), step):
piece = body[i:i + max_chars]
if len(piece) < 40 and i > 0:
continue # drop runts
out.append({"source": f"{doc_name} > {title}", "text": f"{title}. {piece}"})
if i + max_chars >= len(body):
break
return out
emb = TextEmbedding("BAAI/bge-small-en-v1.5")
ce = TextCrossEncoder("Xenova/ms-marco-MiniLM-L-6-v2")
for size in (80, 150, 600):
n = sum(len(chunk(k, v, size, size // 5)) for k, v in DOCS.items())
print(f"max_chars={size:4d} overlap={size//5:3d} -> {n} chunks")
chunks = [c for k, v in DOCS.items() for c in chunk(k, v, 600, 100)]
V = np.array(list(emb.embed([c["text"] for c in chunks])))
V /= np.linalg.norm(V, axis=1, keepdims=True)
def retrieve(q, k=4, rerank_to=2):
qv = next(iter(emb.query_embed(q)))
qv = qv / np.linalg.norm(qv)
sims = V @ qv
top = list(np.argsort(-sims)[:k])
rr = list(ce.rerank(q, [chunks[i]["text"] for i in top]))
order = [top[j] for j in np.argsort(-np.array(rr))][:rerank_to]
return top, order, sims
def stub_llm(question, ctx):
"""STUB, not a real model: returns the context sentence sharing most words with the question."""
qw = set(re.findall(r"\w+", question.lower()))
best = max(((len(qw & set(re.findall(r"\w+", s.lower()))), s, src) for src, t in ctx for s in re.split(r"(?<=\.) ", t)), key=lambda x: x[0])
return f"{best[1]} [{best[2]}]"
tests = [
("Can I get my money back on a monthly plan?", "billing.md > Refunds"),
("How do I stop node_modules from uploading?", "sync.md > Ignoring files"),
("Who can remotely wipe a stolen laptop?", "security.md > Lost devices"),
("What is the maximum file size?", None), # not in the docs
]
for q, gold in tests:
top, order, sims = retrieve(q)
srcs = [chunks[i]["source"] for i in top]
print(f"\nQ: {q}")
print(" top-4 by vector :", [f"{chunks[i]['source']} ({sims[i]:.2f})" for i in top])
print(" after rerank, 2 :", [chunks[i]["source"] for i in order])
if gold is None:
print(" diagnosis : no gold chunk exists; best vector score", f"{sims[top[0]]:.2f}", "-> need a refusal rule")
else:
stage = "OK" if gold in [chunks[i]["source"] for i in order] else ("RERANK DROPPED IT" if gold in srcs else "RETRIEVAL MISS")
print(" diagnosis :", stage)
print(" stub answer :", stub_llm(q, [(chunks[i]["source"], chunks[i]["text"]) for i in order]))
Real output:
max_chars= 80 overlap= 16 -> 14 chunks
max_chars= 150 overlap= 30 -> 9 chunks
max_chars= 600 overlap=120 -> 9 chunks
Q: Can I get my money back on a monthly plan?
top-4 by vector : ['billing.md > Refunds (0.81)', 'billing.md > Cancelling (0.74)', 'billing.md > Changing your card (0.63)', 'security.md > Lost devices (0.61)']
after rerank, 2 : ['billing.md > Refunds', 'billing.md > Cancelling']
diagnosis : OK
stub answer : Annual plans can be refunded in full within 14 days of purchase. [billing.md > Refunds]
Q: How do I stop node_modules from uploading?
top-4 by vector : ['sync.md > Ignoring files (0.61)', 'sync.md > Conflicts (0.57)', 'sync.md > Bandwidth (0.57)', 'security.md > Lost devices (0.55)']
after rerank, 2 : ['sync.md > Bandwidth', 'security.md > Lost devices']
diagnosis : RERANK DROPPED IT
stub answer : Other plans can sign the device out from the web. [security.md > Lost devices]
Q: Who can remotely wipe a stolen laptop?
top-4 by vector : ['security.md > Lost devices (0.74)', 'security.md > Encryption (0.57)', 'sync.md > Conflicts (0.52)', 'security.md > Two-factor sign-in (0.50)']
after rerank, 2 : ['security.md > Lost devices', 'security.md > Two-factor sign-in']
diagnosis : OK
stub answer : Admins on the Business plan can remotely wipe a device. [security.md > Lost devices]
Q: What is the maximum file size?
top-4 by vector : ['sync.md > Bandwidth (0.66)', 'sync.md > Ignoring files (0.60)', 'security.md > Encryption (0.57)', 'sync.md > Conflicts (0.56)']
after rerank, 2 : ['security.md > Encryption', 'sync.md > Ignoring files']
diagnosis : no gold chunk exists; best vector score 0.66 -> need a refusal rule
stub answer : Create a .fernignore file in a folder and list patterns, one per line. [sync.md > Ignoring files]
What did the run teach?
Four questions is a demo, not a benchmark. But each result maps to a failure you will meet.
The reranker hurt once. For the node_modules question, plain vector search put the right section first. The cross-encoder then pushed it out of the final two. The question never says "ignore", and the section never says "node_modules". That is a vocabulary gap, and a general-purpose reranker did not bridge it here. Takeaway: measure with and without. The reranker measurements post does that on 43 labelled queries.
The first version of my chunker had a bug. My first run indexed a heading-only chunk for each document's # Title line, with no body. Those junk chunks took top-k slots. I fixed it with the if not body: continue line you see above. If you chunk by heading, print every chunk once and read them.
The "no answer" question got an answer anyway. "What is the maximum file size?" has no gold chunk. Best vector score was 0.66, while the answerable questions scored 0.61 to 0.81. Those ranges overlap, so a single score cutoff would not have separated them in this tiny set. You need a refusal rule built from your own answerable and unanswerable questions, and probably a prompt that permits "not in the documents" as well.
Chunk size did little on this corpus. Sections here are one to two sentences, so 150 and 600 characters give identical counts. Chunk size only matters when sections are long, so do not tune it on a toy.
How do I tell a retrieval miss from a generation error?
Work in this order. Stop at the first stage that fails.
- Is the answer in the source documents at all? If not, you need a refusal, not a better pipeline.
- Did parsing keep it? Search the parsed text for a distinctive phrase.
- Is it intact in one chunk? Search the chunk list. If split, change boundaries or overlap.
- Is that chunk in the top k? If not, it is a retrieval miss: try keyword plus vector, a different embedding model, or a rewritten query.
- Is it in the top n after reranking? If it was in the candidates but dropped, the reranker is the problem. This was the node_modules case.
- Is it in the prompt that was actually sent? Read the real prompt string. Truncation and formatting bugs live here.
- If it is in the prompt and the answer is still wrong, it is a generation error: stronger instruction, fewer or better-ordered chunks, or a better model.
Log, for each test question: gold chunk found in top-k (yes/no), in the final n (yes/no), answer correct (yes/no). Three columns separate retrieval from generation. RAG evaluation metrics covers the standard metrics once you outgrow a spreadsheet.
Where to host the index
At this scale, nowhere: it is a NumPy array. When you want a persistent store like Qdrant next to your app, a small VPS is enough for modest corpora; the self-hosted RAG stack post walks through one, and which Hostinger plan for AI apps compares tiers. If you want one, Hostinger KVM 2 is the smaller of the two I usually point to (affiliate link). Hetzner, DigitalOcean and Qdrant Cloud are non-affiliate options, and managed stores save you the maintenance.
