Stage 4 of 12 · about 2.9 h

Building with LLM APIs

Call a real model, write prompts as specs, trust the output, and harden every call.

Stage 4 swaps Pocket's engine from the tiny GPT you trained for a hosted model reached through an API. You make a first call and read its tokens, cost, stop reason, and latency. You learn what a system prompt is inside the model and write prompts as specs that carry a note as context, tested against a few questions. You see why sampled output must be validated, and get structured answers you can trust. Then you harden the calls with streaming, timeouts, retries with backoff and jitter, prompt caching, and usage logs. By the end Pocket is a command-line tool that answers questions from all your notes.

Before you start

Stages 0 to 3, or comfortable Python with a terminal and git. This stage is the entry point for developers who want to start with applications: each lesson's recap re-teaches the stage 3 ideas it needs, and you will need your own API key.

When you finish

pocket/stage4/ holds first_call.py, prompt_spec.py, structured.py, and pocket.py, a CLI that streams an answer from all the notes in pocket/notes/, retries transient errors with backoff and jitter, caches the notes prefix, and logs usage for every call to stage4/usage.jsonl.

1. Lesson 1 · 40 min

   Your first model call: tokens, cost, and latency

   Call a hosted model from Python, read its stop_reason and token usage, work out what the call cost at a given rate, and pick a model tier by testing.

2. Lesson 2 · 40 min

   Prompts are specs: context engineering

   Explain what a system prompt is inside the model, write a prompt as a spec that carries the note as context and defines an exact fallback, and check it against a fixed set of test questions.

3. Lesson 3 · 45 min

   Structured output you can trust

   Explain why a model's JSON must be validated, validate it with a Pydantic schema, retry with the error fed back or force the shape with a tool definition, and never execute model output.

4. Lesson 4 · 50 min

   Production-grade calls: streaming, retries, caching

   Stream an answer to the terminal, set timeouts, retry only transient errors with exponential backoff and full jitter, cache a stable prompt prefix and compute what it saves, and log usage for every call.