Stage 7 of 12 · about 2.9 h

Evaluate and ship

Measure Pocket, watch it run, and ship it with evidence.

Replace opinions with numbers. You score Pocket on a 24-question golden set with deterministic checks, calibrate a model judge against hand labels with Cohen's kappa, trace every request to price it and time it with p50 and p95, then diagnose each failure as retrieval or generation, choose the fix, and gate Pocket's launch on thresholds and a readiness checklist.

Before you start

Pocket answers questions from your notes with citations and a fixed "I don't know" sentence through the stage 5 grounded-answer function, and you know the stage 6 guardrails.

When you finish

pocket/stage7/ holds evals.py, judge.py, tracing.py and ship.py: per-category pass rates and refusal precision and recall on a golden set, a judge with a measured kappa, JSON Lines traces with cost and latency, and a SHIP_NOTE.md that records the launch decision. This closes the core path: you have trained a model with gradient descent, built a small transformer, built an LLM app with structured output, retrieval and tools, and you can evaluate, monitor and ship it on evidence. Stages 8 to 11 go deeper: production RAG, agentic patterns, agent testing, and deployment.

1. Lesson 1 · 45 min

   Evals before opinions

   You can score Pocket on a fixed golden set with deterministic checks, report pass rate per category and refusal precision and recall, and tell a real change from noise.

2. Lesson 2 · 45 min

   LLM-as-judge

   You can grade Pocket's answers with a rubric-based model judge, measure its agreement with human labels using raw agreement, Cohen's kappa and the fail catch rate, and test it for position and verbosity bias.

3. Lesson 3 · 40 min

   Observability and cost

   You can write one JSON Lines trace per Pocket request, compute cost per request and per user from token counts, report p50 and p95 latency, and keep note content out of the log.

4. Lesson 4 · 45 min

   Choose the fix, then ship

   You can tell a retrieval failure from a generation failure using the trace, choose between a prompt, retrieval or fine-tuning fix, and gate Pocket's launch on thresholds and a readiness checklist written down in advance.