The stages
In order, from the math toolkit to shipping. Each stage ends with something working in your Pocket repo. Skip any stage you already know.
Stage 0 · 4 lessons · about 2.8 h
The math toolkit
Machine learning rests on four pieces of math, and this track teaches each one in code rather than on paper. You start with how computers store numbers and how numpy arrays let you compute on thousands of them at once. Then you learn what a derivative is and how to check one numerically, how exp and log turn scores into probabilities without overflowing, and how to score a model by the probability it gives to real data. Every lab runs in pocket/stage0/ and some use your own notes as data.
You build: pocket/stage0/ holds four working scripts, ending with probability.py, which fits a letter distribution to your notes and scores it by average negative log-likelihood against a uniform baseline.
Stage 1 · 4 lessons · about 2.7 h
How machines learn
Learning, in machine learning, means choosing a function's parameters so its predictions match data. You start by fitting a line to your own note-review times by hand. Then you move the same model into numpy matrix multiplies and rank Pocket's notes by cosine similarity, give models a loss that measures how wrong they are, and finish by deriving the gradient and letting gradient descent find the parameters for you. Every later stage, from neural networks to language models, reuses these four pieces.
You build: pocket/stage1/ holds four working scripts, ending with fit_line.py, which fits the review-time line by gradient descent, passes a finite-difference gradient check and matches np.polyfit.
Stage 2 · 4 lessons · about 2.8 h
Neural networks from scratch
You start with one neuron and prove why a network needs a nonlinearity to solve even XOR. You then derive the local gradients of each operation and write a small autograd that computes every gradient with the chain rule, use it to train a multilayer network with mini-batches and a validation set, and finish by fighting overfitting and porting the network to PyTorch with proof that both versions agree.
You build: pocket/stage2/ holds neuron.py, tiny_grad.py, train_mlp.py and torch_mlp.py: an autograd you wrote, a tagger trained with it, and a PyTorch port verified to compute the same loss and gradients.
Stage 3 · 4 lessons · about 2.9 h
Language models from scratch
Build a language model the slow way. Turn text into tokens with byte-pair encoding. Count a bigram model, sample from it, and score it with negative log-likelihood, which turns out to be the cross-entropy loss you already know. Work attention through by hand on three tokens, then write one causal head and check it. Finally stack heads into a tiny transformer, name every part and its shape, train it on your notes on a CPU until it beats the bigram, and sample from it. You finish knowing what happens inside an LLM and why a real one is so much larger.
You build: pocket/stage3/ holds a BPE tokenizer, a bigram baseline scoring 2.447 on the sample notes' validation text, an attention head checked against a hand calculation, and a 165,855-parameter GPT trained on your notes that beats the bigram and samples text in their style.
Stage 4 · 4 lessons · about 2.9 h
Building with LLM APIs
Stage 4 swaps Pocket's engine from the tiny GPT you trained for a hosted model reached through an API. You make a first call and read its tokens, cost, stop reason, and latency. You learn what a system prompt is inside the model and write prompts as specs that carry a note as context, tested against a few questions. You see why sampled output must be validated, and get structured answers you can trust. Then you harden the calls with streaming, timeouts, retries with backoff and jitter, prompt caching, and usage logs. By the end Pocket is a command-line tool that answers questions from all your notes.
You build: pocket/stage4/ holds first_call.py, prompt_spec.py, structured.py, and pocket.py, a CLI that streams an answer from all the notes in pocket/notes/, retries transient errors with backoff and jitter, caches the notes prefix, and logs usage for every call to stage4/usage.jsonl.
Stage 5 · 4 lessons · about 2.8 h
Retrieval and RAG
You give Pocket a memory it can search. You embed the notes with a sentence model and rank them by cosine similarity, then split them into labeled chunks that fit the model's input and save an index. Pocket retrieves the best chunks, answers from them with citations your code checks, prefers the newest note when notes disagree, and says it does not know when the notes are silent. Last, you measure retrieval with recall@k and MRR on the sample notes' golden questions and test BM25 and hybrid fusion against that baseline.
You build: pocket/stage5/ holds a 192-chunk index of the sample notes, an answer script that passes six golden tests (two changed-over-time, four unanswerable) with checked citations, and a retrieval report with recall@3 and MRR for dense, BM25 and hybrid search.
Stage 6 · 4 lessons · about 2.9 h
Tools and agents
A model cannot run code, but it can ask your code to. You define tools with JSON Schema, read the model's tool_use blocks, run the functions yourself, and send the results back. Then you wrap that in a loop with a stated invariant, every stop condition handled, and step and token budgets, so Pocket can search, read, and count on its own. Because notes can carry injected instructions, you threat-model the agent and add guards that hold even when the model is fooled: no write tools by default, a path allowlist, and human approval. Last, you serve Pocket's tools over MCP and test them with a client you write.
You build: pocket/stage6/ holds tools.py, agent.py, guarded.py and pocket_mcp.py: an agent that decides when to search and read your notes, stops within set budgets, cannot write without your approval, and serves its note tools to any MCP host.
Stage 7 · 4 lessons · about 2.9 h
Evaluate and ship
Replace opinions with numbers. You score Pocket on a 24-question golden set with deterministic checks, calibrate a model judge against hand labels with Cohen's kappa, trace every request to price it and time it with p50 and p95, then diagnose each failure as retrieval or generation, choose the fix, and gate Pocket's launch on thresholds and a readiness checklist.
You build: pocket/stage7/ holds evals.py, judge.py, tracing.py and ship.py: per-category pass rates and refusal precision and recall on a golden set, a judge with a measured kappa, JSON Lines traces with cost and latency, and a SHIP_NOTE.md that records the launch decision. This closes the core path: you have trained a model with gradient descent, built a small transformer, built an LLM app with structured output, retrieval and tools, and you can evaluate, monitor and ship it on evidence. Stages 8 to 11 go deeper: production RAG, agentic patterns, agent testing, and deployment.
Stage 8 · 4 lessons · about 3.2 h
RAG in production
You take Pocket's retrieval from a working prototype to something you can run for years. You build an IVF index by hand and measure the recall it trades for speed, then decide when brute force is still right. You attach metadata to chunks, see why a post-filter returns fewer than k results, and keep the index fresh with content hashes that re-embed only what changed and rebuild when the model changes. You rewrite follow-up questions, fuse several phrasings, and rerank a short list, reading every gain per question. Last, you assemble context under a budget and score it with context precision and recall, so you can tell a retrieval failure from a generation failure.
You build: pocket/stage8/ holds ivf_index.py (an IVF index with a recall-versus-nprobe table), incremental_index.py (a hash-keyed index that re-embeds only changed chunks), query_and_rerank.py (rewriting, multi-query fusion and reranking scored on the golden questions) and assemble_context.py (a budgeted context assembler with context precision and recall).
Stage 9 · 4 lessons · about 3.2 h
Agentic AI patterns
An agent loop is one pattern among several, and often not the best one. You start with workflows: a voting router and a chain of narrow steps with code gates, so Pocket's common questions run the same way every time at a known cost. Then a planner emits a JSON plan that code validates, workers run the steps with fresh contexts, a failed step triggers one bounded replan, and a critique loop revises the answer against a rubric until it passes or stops improving. You give Pocket memory that survives sessions, written through a policy with sources and dates, and compaction that keeps long conversations under a token budget without breaking the tool protocol. Last, a lead agent dispatches scoped subagents in parallel, merges their reports, and you compute what that costs against one agent. Every lab runs offline on a scripted stand-in model.
You build: pocket/stage9/ holds workflows.py, plan_reflect.py, memory.py and multi_agent.py: a gated router and chain, a planner with workers, replanning, and a capped critique loop, a memory store with compaction, and a lead with scoped subagents whose token cost you have measured.
Stage 10 · 4 lessons · about 3.2 h
Agent development: tests and reliability
An agent that works once is not an agent you can rely on. You test Pocket's loop without a model, using a strict scripted fake and recorded cassettes, so every commit gets the same verdict. You score live runs by outcome and final state, read the path through tool precision, recall, and redundant calls, and report reliability as pass^k. You redesign append_note so wrong calls are invalid, retries happen only where they help, and a lost reply never writes twice. Last, you checkpoint the loop after every step, so a crash or a human approval pause resumes without re-running anything.
You build: pocket/stage10/ holds test_agent.py, trajectory_eval.py, tool_wrapper.py and checkpoint.py: a deterministic test suite for the loop, a trajectory scorer with pass@k and pass^k, an idempotent validated write tool, and a loop that pauses for approval and survives a crash without writing twice.
Stage 11 · 4 lessons · about 3.1 h
Deploying agents
Turn Pocket's agent into a service other people can rely on. You serve it as background jobs with idempotent submission and cancellation, run model-written code in a sandbox and keep secrets out of the model's context, trace every run as spans with budgets, rate limits and a circuit breaker, and release changes through an offline gate, shadow traffic and a hash-bucketed canary that rolls itself back.
You build: pocket/stage11/ holds serve.py, sandbox.py, observe.py and rollout.py: Pocket's agent served as background runs with events, idempotent retries and cancellation, model-written code run under limits with secrets kept in the tool layer, every run traced as spans with per-user rate limits, budgets, a circuit breaker and alerts, and releases rolled out through a canary with an automatic rollback rule and a kill switch. This is the end of the path. You trained a model with gradient descent, built a small transformer, built an LLM app with structured output, added retrieval and made it production RAG, gave it tools, built agents and agentic patterns, tested and evaluated them, and deployed an agent as a service you can watch, limit and roll back.