Embeddings
Embedding geometry: the modality gap, simplex codebooks, the Welch bound, and what trained features spend their dimensions on.
-
Auditing Latent Space Geometry in JAX
A runnable companion to the Welch-bound latent-space post: generate GIFs and implement the JAX metrics that tell you whether embeddings are collapsing, wasting rank, forming a simplex, or pressing against the Welch floor.
-
Organizing Randomness: Contrastive Learning in JAX
A block-by-block JAX + Optax implementation of six contrastive losses, each watched as a real animated GIF turning random 2D points into organized embeddings. The runnable companion to "Untangling the Moons."
-
Untangling the Moons: A Visual History of Contrastive Learning
Eight contrastive losses, twenty years of history, one interactive playground. Watch pair, triplet, InfoNCE, CLIP, SupCon, SigLIP, alignment+uniformity, and cosine→0 organize 2D points, and see which ones know when to stop.
-
Opposite Is Not Different: The Cosine-Similarity Bug in CLIP and Contrastive Learning
Maximum difference between two unit vectors is orthogonality (cos = 0), not opposition (cos = −1). CLIP, InfoNCE, and SimCLR have been optimizing for the wrong target for years.