Latent Space
What makes a good latent space: the simplex, the Welch bound, spectral codebooks, and the information that lives between prototypes.
-
Distillation Is a Geometry, Not an Answer Key
Knowledge distillation has a standing puzzle: Hinton's student recognized 98.6% of the digit 3s in the test set after training on a transfer set with every 3 deleted. An answer key cannot do that, so what actually crosses the wire? This post gives dark knowledge a data type, a class-similarity kernel, and runs the experiment that isolates it: a student trained on nothing but pairwise relations, no labels, no soft targets, no class names, measured against the label-trained ceiling and the random floor. With live experiments: watch the kernel accumulate from single outputs, turn the temperature knob on how much geometry leaks, train a relational student in the page, and watch whose spectrum the student grows into.
-
Latent on the Spectrum, in JAX
A runnable companion to Latent on the Spectrum: build a codebook as the spectral embedding of a label kernel in JAX (classical MDS with square-root eigenvalue scaling), watch a flat spectrum become the simplex and a graded one become the horseshoe, measure kernel-target alignment, split a representation into its between-class prototype frame and within-class information spectrum, and watch neural collapse grind the information to zero.
-
Latent on the Spectrum: Why Cats Sit Closer to Dogs Than to Cars
The regular simplex is the perfect codebook only when classes are strangers, and real labels are not strangers. A latent space is a lossy, finite-dimensional encoding of a label-similarity kernel: the codebook is the top eigenmodes of that kernel, the information rides in the modes below them, and the Welch bound sets the geometry of that channel. A follow-up to the Welch-bound post with live in-browser experiments: steer a codebook from simplex to taxonomy, spend a dimension budget, watch neural collapse grind the information spectrum to zero, read dark knowledge off a wandering feature, and see a structured codebook make better mistakes.
-
Auditing Latent Space Geometry in JAX
A runnable companion to the Welch-bound latent-space post: generate GIFs and implement the JAX metrics that tell you whether embeddings are collapsing, wasting rank, forming a simplex, or pressing against the Welch floor.
-
What Makes a Good Latent Space? The Welch Bound and the Simplex
The hidden codebook inside representation learning: why collapse happens, why opposition is a trap, why class means form a simplex, and why the Welch bound sets the best geometry when too many concepts share too few dimensions.