Stage 5 of 12 · about 2.8 h

Retrieval and RAG

Find the right note, then answer from it with citations.

You give Pocket a memory it can search. You embed the notes with a sentence model and rank them by cosine similarity, then split them into labeled chunks that fit the model's input and save an index. Pocket retrieves the best chunks, answers from them with citations your code checks, prefers the newest note when notes disagree, and says it does not know when the notes are silent. Last, you measure retrieval with recall@k and MRR on the sample notes' golden questions and test BM25 and hybrid fusion against that baseline.

Before you start

You can compute cosine similarity with numpy, explain tokens and what a transformer outputs, and call an LLM API from Python with a system prompt, as built in stages 1, 3 and 4.

When you finish

pocket/stage5/ holds a 192-chunk index of the sample notes, an answer script that passes six golden tests (two changed-over-time, four unanswerable) with checked citations, and a retrieval report with recall@3 and MRR for dense, BM25 and hybrid search.

1. Lesson 1 · 40 min

   Search by meaning

   Embed every note with a sentence model, rank the notes against a question by cosine similarity, and show where this beats keyword search and where it fails.

2. Lesson 2 · 40 min

   Chunking and indexing

   Split notes on headings into overlapping windows that fit the model's input, embed each chunk, and save a vector index as a float32 numpy array plus a JSON list in the same order.

3. Lesson 3 · 45 min

   Grounded answers with citations

   Retrieve the top chunks for a question, send them to Claude with ids, and check in code that the answer cites only chunks it was sent, prefers the newest note when notes disagree, and declines when the notes are silent.

4. Lesson 4 · 45 min

   Evaluating and improving retrieval

   Measure retrieval with recall@k and MRR on a labeled question set, implement BM25 and reciprocal rank fusion, and keep a change only when the numbers improve.