Geometry of Representations
8 parts
Where do representations live, and what makes a latent space good? Contrastive learning, the geometry of embedding spaces, and why the usual activation functions work against it.
All 8 parts →Records of the !mmortal Data Scientist
Notes on AI, math, and the long road. Live slow, die whenever.
Long-form pieces on machine learning interpretability, kernel methods, contrastive learning, and the geometry of neural network representations. Most posts ship with interactive visualisations you can play with in the browser. The posts run in series that build on each other; see the map for how they connect.
Latest
Full archive by date →Series, in reading order
Each series is a narrative: start at part 1.
Attention Is a Kernel
5 parts
Attention, read as kernel regression: what the softmax is really computing, why that makes it explainable, and what happens when you make the kernel cheap.
- Start here Attention is Explainable Because it is a Kernel
- Newest The Kernel Between the Roles
Weights in Kernel Space
5 parts
Once everything is a kernel, what is a weight? An interlude on RKHS foundations: where a weight lives, what it can be, and why the MLP block is a representer theorem.
- Start here The Readout is a Convex Combination of Prototypes
- Newest Why Regularization Is a Price List
The Prototype Network
12 parts · ongoing
Replace the activation with a finite, positive-definite kernel and a network becomes a list of prototypes you can read, edit by hand, and finally collapse into a single fixed-point operator.
- Start here What a Finite Kernel Buys an MLP
- Newest One Kernel, Fitted Twice
Networks as Integrators
5 parts · ongoing
Numerical analysis as an architecture catalog: skip connections as an Euler step, momentum nets as half of Newton, and conservation laws as testable predictions about hidden states.
- Start here Your Skip Connection Is Half of Newton
- Newest Depth on Demand