Stage 2 of 12 · about 2.8 h

Neural networks from scratch

Build a neuron, an autograd and a training loop, then port it to PyTorch.

You start with one neuron and prove why a network needs a nonlinearity to solve even XOR. You then derive the local gradients of each operation and write a small autograd that computes every gradient with the chain rule, use it to train a multilayer network with mini-batches and a validation set, and finish by fighting overfitting and porting the network to PyTorch with proof that both versions agree.

Before you start

You can take a derivative with the chain rule, multiply matrices in numpy, compute MSE, and fit a line with gradient descent, as in stages 0 and 1.

When you finish

pocket/stage2/ holds neuron.py, tiny_grad.py, train_mlp.py and torch_mlp.py: an autograd you wrote, a tagger trained with it, and a PyTorch port verified to compute the same loss and gradients.

1. Lesson 1 · 35 min

   The neuron and nonlinearity

   You can compute a neuron's output by hand, prove that stacked linear layers collapse into one, and solve XOR with a hidden ReLU layer.

2. Lesson 2 · 45 min

   Backpropagation with a tiny autograd

   You can derive the local gradients of +, *, tanh and ReLU, build a scalar autograd that applies the chain rule in reverse topological order, and show why a value used twice collects gradient with +=.

3. Lesson 3 · 40 min

   The training loop

   You can train a small multilayer network with your own autograd, using a fixed train/validation split, shuffled mini-batches, and a correct forward, loss, zero-grad, backward, step loop.

4. Lesson 4 · 45 min

   Generalization, then PyTorch

   You can read training and validation curves, apply weight decay and early stopping, name the PyTorch equivalent of every piece you built, and prove a PyTorch port computes the same loss and gradients.