Your path

The route in text

1. Stage 0

   The math toolkit

   You build: pocket/stage0/ holds four working scripts, ending with probability.py, which fits a letter distribution to your notes and scores it by average negative log-likelihood against a uniform baseline.

   • Numbers and arrays40 min
   • Slopes and derivatives40 min
   • Exponentials and logs40 min
   • Probability basics45 min

2. Stage 1

   How machines learn

   You build: pocket/stage1/ holds four working scripts, ending with fit_line.py, which fits the review-time line by gradient descent, passes a finite-difference gradient check and matches np.polyfit.

   • A model is a function with parameters30 min
   • Vectors and matrices in numpy40 min
   • Measuring wrong: loss functions45 min
   • Gradient descent45 min

3. Stage 2

   Neural networks from scratch

   You build: pocket/stage2/ holds neuron.py, tiny_grad.py, train_mlp.py and torch_mlp.py: an autograd you wrote, a tagger trained with it, and a PyTorch port verified to compute the same loss and gradients.

   • The neuron and nonlinearity35 min
   • Backpropagation with a tiny autograd45 min
   • The training loop40 min
   • Generalization, then PyTorch45 min

4. Stage 3

   Language models from scratch

   You build: pocket/stage3/ holds a BPE tokenizer, a bigram baseline scoring 2.447 on the sample notes' validation text, an attention head checked against a hand calculation, and a 165,855-parameter GPT trained on your notes that beats the bigram and samples text in their style.

   • Text to tokens40 min
   • The bigram model45 min
   • Embeddings and self-attention45 min
   • A tiny transformer45 min

5. Stage 4

   Building with LLM APIs

   You build: pocket/stage4/ holds first_call.py, prompt_spec.py, structured.py, and pocket.py, a CLI that streams an answer from all the notes in pocket/notes/, retries transient errors with backoff and jitter, caches the notes prefix, and logs usage for every call to stage4/usage.jsonl.

   • Your first model call: tokens, cost, and latency40 min
   • Prompts are specs: context engineering40 min
   • Structured output you can trust45 min
   • Production-grade calls: streaming, retries, caching50 min

6. Stage 5

   Retrieval and RAG

   You build: pocket/stage5/ holds a 192-chunk index of the sample notes, an answer script that passes six golden tests (two changed-over-time, four unanswerable) with checked citations, and a retrieval report with recall@3 and MRR for dense, BM25 and hybrid search.

   • Search by meaning40 min
   • Chunking and indexing40 min
   • Grounded answers with citations45 min
   • Evaluating and improving retrieval45 min

7. Stage 6

   Tools and agents

   You build: pocket/stage6/ holds tools.py, agent.py, guarded.py and pocket_mcp.py: an agent that decides when to search and read your notes, stops within set budgets, cannot write without your approval, and serves its note tools to any MCP host.

   • Tool calling: the model asks, your code acts40 min
   • The agent loop45 min
   • Guardrails: data is not instructions45 min
   • MCP: tools any client can use45 min

8. Stage 7

   Evaluate and ship

   You build: pocket/stage7/ holds evals.py, judge.py, tracing.py and ship.py: per-category pass rates and refusal precision and recall on a golden set, a judge with a measured kappa, JSON Lines traces with cost and latency, and a SHIP_NOTE.md that records the launch decision. This closes the core path: you have trained a model with gradient descent, built a small transformer, built an LLM app with structured output, retrieval and tools, and you can evaluate, monitor and ship it on evidence. Stages 8 to 11 go deeper: production RAG, agentic patterns, agent testing, and deployment.

   • Evals before opinions45 min
   • LLM-as-judge45 min
   • Observability and cost40 min
   • Choose the fix, then ship45 min

9. Stage 8

   RAG in production

   You build: pocket/stage8/ holds ivf_index.py (an IVF index with a recall-versus-nprobe table), incremental_index.py (a hash-keyed index that re-embeds only changed chunks), query_and_rerank.py (rewriting, multi-query fusion and reranking scored on the golden questions) and assemble_context.py (a budgeted context assembler with context precision and recall).

   • Vector indexes: exact, IVF and HNSW45 min
   • Metadata, filters and a fresh index45 min
   • Query rewriting, fusion and reranking50 min
   • Context assembly and end-to-end RAG evals50 min

10. Stage 9

    Agentic AI patterns

    You build: pocket/stage9/ holds workflows.py, plan_reflect.py, memory.py and multi_agent.py: a gated router and chain, a planner with workers, replanning, and a capped critique loop, a memory store with compaction, and a lead with scoped subagents whose token cost you have measured.

    • Workflows before agents45 min
    • Planning, replanning, and reflection50 min
    • Memory and context45 min
    • Multi-agent systems50 min

11. Stage 10

    Agent development: tests and reliability

    You build: pocket/stage10/ holds test_agent.py, trajectory_eval.py, tool_wrapper.py and checkpoint.py: a deterministic test suite for the loop, a trajectory scorer with pass@k and pass^k, an idempotent validated write tool, and a loop that pauses for approval and survives a crash without writing twice.

    • Deterministic tests for a nondeterministic agent45 min
    • Trajectory evals and pass^k45 min
    • Tool design and error recovery50 min
    • Checkpoints, resume, and approval pauses50 min

12. Stage 11

    Deploying agents

    You build: pocket/stage11/ holds serve.py, sandbox.py, observe.py and rollout.py: Pocket's agent served as background runs with events, idempotent retries and cancellation, model-written code run under limits with secrets kept in the tool layer, every run traced as spans with per-user rate limits, budgets, a circuit breaker and alerts, and releases rolled out through a canary with an automatic rollback rule and a kill switch. This is the end of the path. You trained a model with gradient descent, built a small transformer, built an LLM app with structured output, added retrieval and made it production RAG, gave it tools, built agents and agentic patterns, tested and evaluated them, and deployed an agent as a service you can watch, limit and roll back.

    • Serving an agent: runs, workers and events45 min
    • Sandboxes and least privilege45 min
    • Production observability and cost control45 min
    • Rollout and operations50 min