series · 6 parts · in progress
Geometry of Representations
Where do representations live, and what makes a latent space good? Contrastive learning, the geometry of embedding spaces, and why the usual activation functions work against it.
Start reading → What Activations Do to Geometry- 01 What Activations Do to Geometry ReLU, GELU, and their relatives enter a layer's Jacobian as an input-dependent row scaling. Here is when that scaling erases directions, when it merely distorts them, and what the usual repairs actually guarantee.
- 02 Untangling the Moons: A Visual History of Contrastive Learning Eight contrastive losses, twenty years of history, and one geometric audit. Watch the losses organize the same 2D points while separating opposition, orthogonality, simplex packing, and statistical independence. JAX companion Organizing Randomness: Contrastive Learning in JAX
- 03 What Makes a Good Latent Space? The Welch Bound and the Simplex The hidden codebook inside representation learning: why collapse happens, why opposition is a trap, why class means form a simplex, and why the Welch bound sets the best geometry when too many concepts share too few dimensions. JAX companion Auditing Latent Space Geometry in JAX
- 04 Latent on the Spectrum: Why Cats Sit Closer to Dogs Than to Cars A label-similarity kernel can be turned into a target codebook by spectral embedding: retain its leading eigenmodes, scale by their square roots, and spend a finite dimension budget. Interactive experiments move that designed geometry from a simplex toward a taxonomy, then compare it with the class-mean and within-class spectra measured in trained representations. JAX companion Latent on the Spectrum, in JAX
- 05 The Three States of Information In these training runs, representation geometry moves through three recognizable regimes: random, organized into local clusters, and globally structured around separated class means. Interactive experiments test when loss plateaus coincide with those reorganizations—and when schedules change the order. JAX companion The Three States of Information, in JAX
- 06 Distillation Is a Geometry, Not an Answer Key What crosses the wire in knowledge distillation besides the winning class? This experiment extracts a class-similarity kernel from teacher outputs and trains a student on pairwise relations alone—no labels, class names, or target probabilities. On Fashion-MNIST, the student recovers much of the label-trained geometry and approaches the spectrum of the transferred relation matrix. JAX companion Distillation as Kernel Transfer, in JAX/Flax NNX
This series is still being written. New parts land in the archive and the RSS feed as they publish.