What I’m learning

What I’m learning#

A running log of ML research I've been thinking about — papers I'm reading, ideas I'm turning over. Every entry is a live notebook.

A stylised loss landscape with a descent trajectory.

2026

  1. Jul 26 ML

    Sparse autoencoders from scratch

    Part 1 rebuilt from zero background: linear AEs = PCA, ReLU superposition, the cat/dog/French/CAPS round trip, then a trained ReLU SAE verified against a known dictionary with L0/FVU metrics.
  2. Jul 25 ML

    Training your own JumpReLU SAE on Gemma 3 1B

    End-to-end: collect layer-13 residual activations from The Pile, define and train a 16k-latent JumpReLU SAE from scratch (threshold LR trick, resampling, L0 targeting), evaluate with loss-recovered, compare to the official GemmaScope-2 SAE, and read off the monosemantic features it found. Part 4 of the SAE speedrun.
  3. Jul 25 ML

    SAE architectures & JumpReLU

    The L1 shrinkage problem, then TopK / Gated / JumpReLU — implementing JumpReLU's straight-through-estimator gradients by hand and showing it removes shrinkage. The architecture GemmaScope uses. Part 2 of the SAE speedrun.
  4. Jul 25 ML

    Reconstructing a Neuronpedia dashboard

    Loading the real GemmaScope-2 SAE for Gemma 3 1B and rebuilding every panel of the latent-7447 'fruit' dashboard from scratch: top logits, logits histogram, activation density, max-activating-examples heatmap, cosine neighbours. Part 3 of the SAE speedrun.
  5. May 04 PAPER

    Influence functions: where does the Hessian come from?

    A derivation of the Koh & Liang (2017) influence function from the implicit function theorem, verified numerically on a small MLP.
  6. Apr 24 ML

    Forward, backward, and hooks in PyTorch

    What actually happens when you call .backward(), and how register_forward_hook / register_full_backward_hook let you capture per-layer (a, δ) factors.

Adding a new entry

Add YAML front-matter to any content page with kb_date, kb_tag, kb_title, and kb_desc fields. The timeline above is auto-generated at build time from all pages that declare these fields — no need to edit this file.

Example front-matter:

---
kb_date: "2026-05-04"
kb_tag: paper
kb_title: "My new page"
kb_desc: "A short description for the timeline card."
---

Tag values: ai, ml, maths, paper, note.