Stage 10 of 12 · about 3.2 h

Agent development: tests and reliability

Test the loop exactly, measure the model honestly, and make every tool call safe to retry and resume.

An agent that works once is not an agent you can rely on. You test Pocket's loop without a model, using a strict scripted fake and recorded cassettes, so every commit gets the same verdict. You score live runs by outcome and final state, read the path through tool precision, recall, and redundant calls, and report reliability as pass^k. You redesign append_note so wrong calls are invalid, retries happen only where they help, and a lost reply never writes twice. Last, you checkpoint the loop after every step, so a crash or a human approval pause resumes without re-running anything.

Before you start

You can write a bounded agent loop with tool_use and tool_result, guard its write tools with approval, score a golden set, and retry transient API errors with backoff, as in stages 4, 6, and 7.

When you finish

pocket/stage10/ holds test_agent.py, trajectory_eval.py, tool_wrapper.py and checkpoint.py: a deterministic test suite for the loop, a trajectory scorer with pass@k and pass^k, an idempotent validated write tool, and a loop that pauses for approval and survives a crash without writing twice.

1. Lesson 1 · 45 min

   Deterministic tests for a nondeterministic agent

   You can test an agent loop without calling a model: unit-test its tools, script a fake model that is as strict as the real API, record and replay real responses by request hash, and check budgets and the loop invariant, keeping model quality for the eval suite.

2. Lesson 2 · 45 min

   Trajectory evals and pass^k

   You can grade agent runs by outcome and final state, measure the path with tool precision, recall, redundant calls, steps, and budget hits, and report reliability over repeated runs with pass@k and pass^k computed by hand.

3. Lesson 3 · 50 min

   Tool design and error recovery

   You can design a tool for a model to use (clear name, description, and a schema that makes wrong input impossible), validate its input with errors that name the fix, retry only transient failures, make writes idempotent with a key, and cap large results with a pointer to the rest.

4. Lesson 4 · 50 min

   Checkpoints, resume, and approval pauses

   You can persist an agent's full state after every model reply and every tool result with an atomic write, resume after a crash without re-calling the model or re-applying a side effect, and turn a human approval into a saved state that resumes later.