Stage 3 of 12 · about 2.9 h

Language models from scratch

Tokens, a bigram baseline, attention by hand, and a tiny GPT trained on your own notes.

Build a language model the slow way. Turn text into tokens with byte-pair encoding. Count a bigram model, sample from it, and score it with negative log-likelihood, which turns out to be the cross-entropy loss you already know. Work attention through by hand on three tokens, then write one causal head and check it. Finally stack heads into a tiny transformer, name every part and its shape, train it on your notes on a CPU until it beats the bigram, and sample from it. You finish knowing what happens inside an LLM and why a real one is so much larger.

Before you start

You can write a PyTorch training loop with a train and validation split, read an overfitting curve, and compute softmax and cross-entropy from logits, as built in stages 0 to 2.

When you finish

pocket/stage3/ holds a BPE tokenizer, a bigram baseline scoring 2.447 on the sample notes' validation text, an attention head checked against a hand calculation, and a 165,855-parameter GPT trained on your notes that beats the bigram and samples text in their style.

1. Lesson 1 · 40 min

   Text to tokens

   Turn a folder of notes into token ids with byte-pair encoding, encode and decode new text with the learned merges, and explain why token counts, not character counts, set what a model can read and what it costs.

2. Lesson 2 · 45 min

   The bigram model

   Build a character bigram model from counts, sample text from it, score it with average negative log-likelihood on held-out notes, and show that this score is exactly the cross-entropy loss from stage 1.

3. Lesson 3 · 45 min

   Embeddings and self-attention

   Compute causal scaled dot-product attention by hand on three tokens, implement it as a PyTorch head, and explain what the embedding table, queries, keys, values, scale and mask each do.

4. Lesson 4 · 45 min

   A tiny transformer

   Name every component of a GPT and its tensor shape, train a tiny one on your notes on a CPU until it beats the bigram baseline, sample it with temperature and top-k, and explain what separates it from a real LLM.