Day 3 showed you how to direct the model. Day 4 shows you how to feed it — chunking, embeddings, vector search, re-ranking. The plumbing behind every AI product that knows something private.
A model only knows what it was trained on. Your data was not in that pile.
Your internal wiki, last night's support tickets, the contract signed this morning — none of it exists inside the weights. There are exactly three ways to change that, and only one of them keeps up with a corpus that changes.
Long context did not kill retrieval. Gemini 1.5 Pro shipped a one-million-token window in 2024 and frontier windows have kept growing since. The window stopped being the constraint. The bill did not.
Index your documents once. At query time, find the handful of passages that actually relate to the question, and put only those in the prompt.
Large or changing corpora, per-user permissions, anything that needs a source link. Which is most real products.
"Fine-tuning changes how the model behaves. Retrieval changes what it knows. Confusing the two is the most expensive mistake in this field."
You do not retrieve documents. You retrieve pieces. Where you cut decides what you can find.
A chunk is the unit of retrieval — the smallest thing your system can hand to the model. Cut a sentence in half and neither half is findable: one has the subject, the other has the number. Nothing downstream can repair that. Not a better embedding model, not a re-ranker, not a cleverer prompt.
There is no research-blessed chunk size. It depends on your documents and the questions people actually ask. Common practice sits somewhere around 200–800 tokens with 10–20% overlap — a starting point, not a finding.
Cheapest to implement. Cuts wherever the character count lands — mid-word, mid-sentence, mid-table.
Search stopped being about words. It is geometry now.
Day 2 showed you embeddings — text turned into a list of numbers, a coordinate in a space where meaning is direction. Retrieval is the payoff. Embed every chunk once. Embed the question at query time. The chunks pointing the same way are your answer.
Reciprocal Rank Fusion — Cormack et al. (2009), with k = 60. It reads positions, never scores, so a BM25 score and a cosine score never have to be made comparable. No training, no tuning, roughly ten lines of code.
Dense retrieval reads meaning. This is the query type it was built for.
Retrieval finds fifty maybes. Re-ranking finds the five that answer the question.
Vector search is optimised for recall — get the right chunk somewhere in the top fifty, fast. That is a different job from putting it at number one. So you run a second, more expensive model over the shortlist, and only the shortlist.
Query and chunk are encoded separately. Chunk vectors are computed once, at index time, and never again.
Query and chunk go through the model together, attending to each other token by token, and one relevance score comes out.
Reimers & Gurevych (2019) measured it: finding the most similar pair among 10,000 sentences takes roughly 65 hours with a BERT cross-encoder and about 5 seconds once the sentences are pre-encoded as vectors.
Three stages run once. Three run on every single query.
That split is the whole architecture. Parsing, chunking and indexing are a batch job you run when documents change. Retrieving, re-ranking and generating happen while someone waits — and every millisecond and every token there is on your bill.
Split on the document's own seams, pack to a budget, overlap the boundaries, and prefix each chunk with the title and section it came from.
chunks = split(text, on=["\n## ", "\n\n", ". "], budget=500, overlap=60)
A clause is severed across two chunks. A chunk full of "it" and "those" with the subject three chunks back.
The answer is definitely in the corpus, and it never appears in the top-k.
When a RAG system answers badly, log the retrieved chunks before you touch the prompt. If the answer was not in what you retrieved, it is a retrieval bug, and no amount of prompt engineering will fix it. If it was in there and the model still got it wrong, now you have a generation problem.
RAG reduces hallucination. It does not end it. A model handed the right passage can still misread it, blend it with something it half-remembers from training, or answer a question the passage never addressed. Grounding is an instruction plus an evaluation loop — never a guarantee.
All from published research. All directly applicable to the pipeline you ship next week.
Lewis et al. (2020), "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Facebook AI Research, NeurIPS 2020), is where the term comes from. Its memory was a Wikipedia dump split into 21 million 100-word passages, retrieved with DPR and passed to a BART generator — with the query encoder and generator trained jointly.
Karpukhin et al. (2020), "Dense Passage Retrieval for Open-Domain Question Answering", reported that their dense retriever outperformed a strong Lucene-BM25 system by 9–19% absolute in top-20 passage retrieval accuracy across open-domain QA benchmarks.
Thakur et al. (2021) built BEIR to test retrievers across 18 datasets they were not trained on. BM25 held up as a remarkably strong baseline: several dense models that dominate in-domain fell behind it once the domain shifted. The strongest overall configuration was BM25 retrieval followed by a cross-encoder re-ranker.
Cormack et al. (2009), "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods" (SIGIR), scores each document as the sum of 1/(k + rank) across every ranked list it appears in, with k = 60. It beat both the individual rankers and more sophisticated fusion methods.
Reimers & Gurevych (2019), "Sentence-BERT", noted that finding the most similar pair among 10,000 sentences requires about 50 million BERT cross-encoder inferences — roughly 65 hours on the GPU they used. Pre-encoding each sentence once and comparing vectors brings the same task down to about 5 seconds.
Production vector search runs on approximate nearest neighbour indexes — most commonly HNSW (Malkov & Yashunin), which walks a layered graph instead of scanning every vector. It is approximate by construction: the true best match can be missed, and how often that happens depends on parameters you set.
Day 5 treats retrieval as one tool an agent can reach for. You want to have felt it break first.
Build the smallest RAG that works, over ten of your own documents. Parse, chunk, embed, retrieve the top five, answer from them. No framework, no vector database.
At ten documents you do not need an ANN index — cosine similarity over an array is exact and instant. Skipping it removes a variable while you learn the rest.
Write twenty real questions and, for each one, the chunk that should answer it. Measure recall@5. Then add overlap, then add keyword search, then add a re-ranker — and re-measure after each.
Your first number will be worse than you expect. That is the point: you now have a dial you can turn instead of a vibe you can argue about.