AgentLu
← Learn

Knowledge & Retrieval

16 min read

A model knows what it was trained on and nothing about your school, your handbook, or the email somebody sent this morning. Ask it about your deadline and it will produce something plausible, which is the most dangerous kind of wrong answer there is. Retrieval is how you fix that: find the passages that matter, and put them in front of the model.

The surprising part is how little of this is about the model. Retrieval is its own pipeline with its own failure modes - a corpus, a chunking rule, a ranking rule, a shortlist - and the answer is the last step, not the first. When an assistant gets something wrong, the useful question is almost never 'which model?' but 'did the passage that answers this ever reach it?'

So this chapter is about that pipeline: what a passage is, how to rank one against a question, what to do when two documents disagree, and how to tell the two kinds of failure apart. Everything here is checkable, which is why the exercises show you the query, the passages and their scores, and the ids the answer stood on.

Retrieval is a separate job from answering

The instinct is to paste the whole handbook into the prompt and let the model sort it out. That works for two pages and collapses at twenty: the model has to find the relevant sentence among thousands of irrelevant ones, the cost and the latency grow with every document you add, and - worst of all - the failure is invisible. A model that missed the relevant paragraph answers confidently from the rest of the text.

Splitting the job fixes all three. Your code decides *what* the model sees, using a rule you can read and test. The prompt stays short and stable. And when the answer is wrong you have a place to look: was the answering passage in the shortlist, or was it not?

This is why the exercises hand you a retriever of your own. A model is a poor search engine and a good writer; a term-frequency ranking function is a poor writer and an excellent search engine. Use each for the half it is good at.

const documents = corpus();                       // 1. what you know
const passages = chunk(documents);                // 2. what a passage is
const shortlist = search(passages, question, 3);   // 3. what is relevant
const answer = await llm({ system, user: render(shortlist) });   // 4. what to say
// 5. cite: map the ids the model returned back to real passagesjavascript

The shape of the pipeline. Only the fourth line involves a model - and by then the hard part is already done.

Chunking decides what a score means

A retriever does not score documents, it scores passages. Chunking is the step that decides what one of those is, and it changes everything downstream: chunk too coarsely and a score tells you a long document is vaguely about your topic, which you already knew; chunk too finely and a passage stops making sense on its own, so a correct match is unusable as evidence.

The practical rule is one idea per passage. If a passage can answer two unrelated questions it is too big; if it needs the next passage to mean anything it is too small. Headings and blank lines are the best boundaries you have, because whoever wrote the document already did the grouping for you.

The other half of chunking is overlap: a sentence that straddles two passages can be lost by both. One sentence of overlap is usually enough, and it is worth the duplication - a missing sentence costs far more than a repeated one.

Ranking, and why returning everything is not ranking

A retriever that returns every passage has ranked nothing: it has moved the search problem back into the prompt, which is exactly what you were trying to avoid. The job is to put the *best* three passages in front of the model, which means scoring them and sorting them.

The classic score is term overlap with a twist. Every query word is looked up in every passage, but a word that appears in almost every passage tells you nothing - 'school', 'policy', 'student' - while a word that appears in one passage is the reason to pick that passage. Dividing by how many passages contain the term is the whole trick, and it is why 'wifi' outranks 'password' in a corpus that mentions passwords everywhere.

Then you take the top k. k is a budget: too small and you miss the second passage a question needs, too large and you dilute the prompt with passages that merely share a word. Three to five is a good starting point, and the exercises make you change it once you meet a question that needs two sources.

Asking for a grounded answer

Retrieval gets the right passages into the prompt. The instruction is what stops the model from wandering off them - and what makes the answer checkable afterwards.

An invitation to improvise

Weak
You are a school assistant. Answer the student's question about deadlines.

QUESTION: When is the CS 101 project due?javascript
Improved
You are a school assistant. Answer only from the passages you are given. If they do not answer the question, say so.

QUESTION: When is the CS 101 project due?
PASSAGES: [1] id=syllabus-deadlines#1 ...javascript

The first version invites the model to answer from memory, and a plausible date from memory is worse than no answer at all - it looks like a fact. The second names the evidence, bounds the answer to it, and gives the model a refusal that is a correct outcome rather than a failure.

An answer you cannot check

Weak
Reply with one or two sentences.javascript
Improved
Reply with JSON only: { "answer": "<one or two sentences>", "citations": ["<passage id>"] }javascript

Without the citation field you have a sentence you have to trust. With it, your code can check that every id exists in the shortlist it sent - and a citation that names a passage you never retrieved is caught immediately, because the ids are yours, not the model's.

Building the syllabus assistant, step by step

The exercise's corpus is eight school documents, and the questions are the ones a student actually asks. The starter's retriever is deliberately naive, so every step below is a fix you can watch land.

1. Look at what the starter retrieves

Run "When is the CS 101 project due?" and read the card. The starter chunks one passage per document and 'ranks' by returning the first three documents in order, so the shortlist is attendance, exams and library - three documents that share a word or two with the question and none that answer it. The diagnosis badge says *retrieval*, and names the passage that was missed. Nothing is broken; the retriever has not ranked anything.

2. Chunk before you score

Fix chunk() first. Split each document on blank lines and pack the paragraphs into passages of roughly MIN_CHARS. That single change is what makes a score mean something: now the passage about the CS 101 deadline is its own passage, competing only with passages that are also about deadlines.

const paragraphs = document.text.split(/\n\s*\n/).filter(Boolean);
// group paragraphs until the passage is big enough, then start a new onejavascript

3. Score, and prefer the rare word

Now rank. Tokenize the question, drop the stopwords, and score each passage by the terms it contains - weighting each term by how rare it is in the corpus. 'the' and 'project' both match something, but only one of them tells you where to look. Take the top three and log them; the console shows the passage count and the diagnosis on every run.

const df = passages.filter((other) => text(other).includes(term)).length;
score += Math.log(1 + passages.length / Math.max(1, df));javascript

4. Ask for the citations, and map them back

The starter's prompt asks for an answer and nothing else, so a correct answer still arrives unsourced - and the badge reads *generation*, which is exactly the point of the two-part diagnosis. Add the citations field to the prompt, and keep the filter that drops any id the model was not given. Then the card shows the answering passage with a 'cited' badge, the answer carries its id, and the badge turns green.

5. Add your own documents

The corpus is a file you own (docs.js), and there is an upload slot (data/documents.txt) for sources you would rather not paste into code. Upload a CSV of your own deadlines, ask about one, and watch it get chunked, scored and cited like any other document - or watch the badge tell you which half to fix when it does not.

Which source wins?

Similarity is not the only thing that decides whether a passage should be used. Schools change their minds, and the old document stays in the drive: the 2024 policy that allowed phones in the courtyard is still a perfect lexical match for 'are phones allowed at break?'. A retriever that ranks by similarity alone will happily hand you a rule that no longer exists.

So a shortlist needs a second signal. Recency is the obvious one - an updated date on each document, and a tie-break that prefers the newer one. Authority is another: a policy beats a bulletin that quotes it, even when the bulletin is newer. And supersession is a fact worth recording: the replaced document is still evidence of what used to be true, so you keep it, do not cite it as current, and say which version you used.

That last part is a writing instruction as much as a retrieval one. 'The current version says...' is the difference between an answer a reader can trust and one they have to go and check - and it is the one line that makes a conflicting-documents assistant worth having.

Two failures, two fixes

When a grounded assistant gets something wrong there are only two possibilities, and they could not be more different. Either the passage that answers the question never reached the model - a *retrieval* failure - or it did, and the answer ignored it - a *generation* failure. Fixing the first means changing chunking, scoring or k. Fixing the second means changing the prompt, the citation mapping, or the model. Guessing which one you have is how you spend an afternoon tuning a retriever that was never the problem.

The way to tell them apart is to check the shortlist against a known answer. Keep a small list of questions with the passage that answers each one - an answer key, not a test - and have the run compare. If the answering passage is missing from the shortlist, say so, and say which passages beat it. If it is there and the answer did not cite it, say that instead. The exercises call this diagnose(), and it is the most useful twenty lines in the whole project.

It also changes what you do about a failure. A retrieval failure is a bug you can fix in code, deterministically: the ranking function either finds the passage or it does not, and you can re-run it as often as you like. A generation failure is a prompt problem, which is fuzzier - but at least you know the evidence was in front of the model, which rules out half the search space.

Mistakes worth catching early

What goes wrongWhat it looks likeWhat to do instead
Chunking too coarselyOne passage per document, and every score is the same.Split on headings and blank lines, and cap a passage's size.
Returning everythingThe prompt is long, the model picks the wrong paragraph, and nothing is ranked.Score, sort, and take the top three to five.
Treating every term as equal'School' and 'student' match everything, so everything scores.Weight terms by how rare they are across the corpus.
k too smallA question that needs two passages is answered from one of them.Raise k and re-run: the card shows exactly which passages you were given.
Trusting the citationThe model cites an id you never retrieved.Filter citations against the ids you sent - the ids are yours, not the model's.
Ignoring the document's dateA superseded policy answers a question about the current rule.Carry an updated date, prefer the newer source, and name the version in the answer.
Answering anywayA confident answer to a question the documents never mention.Give the model an explicit refusal, and check it against the shortlist.

Debugging a retrieval pipeline

  • Read the query first. The retriever can only be as good as the words it was given, and a question full of pronouns - 'what about that one?' - has almost nothing to match on.
  • Look at the scores, not just the order. If the top three all score 2.1, the ranking is not separating anything, and you are back to document order with extra steps.
  • Find the passage that should have won, and ask what it has that the winner does not. That answer is your scoring rule, nine times out of ten.
  • Check the chunk boundaries when a passage is almost right. A sentence cut in half by a chunk boundary is invisible to a term match.
  • Run the question that needs two passages, then change k and watch the card. This is the cheapest experiment in the whole exercise, and it teaches the most.
  • When the answer is wrong but the shortlist was right, stop touching the retriever. The evidence was in front of the model; the fix is in the prompt.