Distillation Is a Geometry, Not an Answer Key

The capstone of the latent-space arc. Latent on the Spectrum argued that a latent space is the spectral embedding of a similarity kernel over classes, and its companion showed that a teacher’s softened softmax carries exactly that kernel. This post closes the loop with an experiment you can rerun: strip distillation down to the kernel alone, no labels anywhere, and measure how much of the teacher’s competence survives the handoff.

In 2015, Hinton, Vinyals and Dean reported something that should still bother you. They trained a small network to imitate a big one’s softened outputs on a transfer set from which every example of the digit 3 had been deleted. The student never saw a 3, was never told 3s exist, and its bias for the class had to be fixed up by hand. It then recognized 98.6% of the 3s in the test set.

An answer key cannot do that. If distillation were the student copying the teacher’s answers, deleting a class from the transfer set would delete the class from the student. Something else came over the wire, something that was present in the teacher’s answers about other digits. Hinton called it dark knowledge and pointed at the small probabilities: a 7 that is a little bit 1, a 2 that is faintly a 3. But “small probabilities” is a description, not a mechanism. What object, exactly, is being transferred?

This series has already met the object. It just arrived from the other direction.

Every answer leaks the relations#

What does one softened output actually contain? Take a trained classifier on Fashion-MNIST (the teacher for everything below: a small CNN at 87.2% test accuracy, from scripts/kernel_distill.py). Show it one shirt. At temperature $T=4$ its softmax does not say “Shirt.” It says: mostly Shirt, quite a bit T-shirt, some Pullover, some Coat, essentially zero Trouser, zero Bag. One prediction, and it has already told you how an entire neighbourhood of the class system is arranged from where this example sits.

Now let the examples stream. Each output $p = \mathrm{softmax}(z/T)$ is one noisy receipt of resemblance; average their outer products and the noise cancels while the structure compounds:

$S_{cc'}$ is large exactly when the teacher routinely hesitates between $c$ and $c'$, and hesitation is not a defect here; it is the signal. Watch the estimate assemble from single outputs:

A matrix of pairwise resemblances between classes is a similarity kernel, and the previous post spent its whole length on what such a kernel is worth: a latent space is nothing but the spectral embedding of one. If that phrase is not yet in your bones, one felt image is enough to carry you through this post: diagonalize the similarity matrix and it hands you coordinates, place each class at its row of the top eigenvectors, and the dot products between those positions reproduce the similarities you started from; Latent on the Spectrum builds the rest. So the teacher’s outputs are not answers with some fuzz on them. They are a serialization of the teacher’s class geometry, one example-sized packet at a time. Dark knowledge has a data type, and the data type is a kernel.

The knob on the leak#

If the outputs carry a kernel, the distillation temperature stops being a mysterious hyperparameter and becomes something you can watch. Cold softmax, nearly one-hot outputs, nearly no off-answer mass: the kernel degenerates toward the identity, classes are strangers, and almost nothing about their relations survives into $S$. Warm softmax: the relations stand out. Too hot: everything resembles everything, and the kernel washes toward uniform, which carries no information either.

Drag the dial. The matrix below is recomputed live from the teacher’s real logits at whatever $T$ you choose:

In the run’s numbers: the mean off-diagonal of $S$ is 0.032 at $T=1$, 0.298 at $T=4$, and 0.902 at $T=16$, while the kernel’s effective rank falls from 8.9 to 7.0 to 4.3. The choice $T=4$ is not cosmetic. It selects which kernel the teacher hands over, a genuinely different geometry from the cold one, and everything the student inherits below is downstream of that choice.

Strip distillation to the kernel alone#

Here is the claim this picture forces. If what distillation transfers is the kernel, then the labels, the soft targets, even the per-class structure of the teacher’s output should all be optional. A student trained on nothing but pairwise relations should still inherit a working geometry.

That is a falsifiable statement, so scripts/kernel_distill.py runs it. Five networks, same data, one measuring stick:

1. Teacher: the CNN above, trained on labels.
2. Soft targets (KD): a smaller student, standard distillation, KL to the teacher’s $T=4$ outputs, no hard labels.
3. Relations only: the same smaller student, trained with no labels, no soft targets, no per-class anything. For each batch, the teacher’s softened outputs define a similarity for every pair, and the student’s only job is to make its embedding geometry agree:

where $\hat e$ are the student’s normalized embeddings and $\hat p$ the teacher’s normalized soft outputs. The loss never mentions a class. It only ever says: these two should be this similar.

4. Random: the untrained student, probed as-is. The floor.
5. True labels: the same student architecture trained on the answer key. The ceiling.

Since the relational student has no classifier head at all, competence is measured the same way for everyone: freeze the network, then probe its embedding with the labels (a linear probe, and a nearest-centroid probe). The labels appear only in the measuring stick, never in the student’s gradient.

Before the numbers, feel the mechanism. This panel trains a real relational student in your browser, by gradient descent, on the actual target Gram the run’s teacher assigned to thirty garments:

Where a label-free student lands#

So how much of the teacher’s competence comes through a channel that thin? The expectation is “somewhere between random and labels,” and where it lands is the measurement of how much of classification knowledge is geometry rather than answers.

The numbers, from the run (seeds 0, 1, 2 for the relational student; seed 0 elsewhere):

Start at the floor, because it is high: a random CNN’s features already probe at 77.5%, the same random-features effect the Arc C posts kept meeting, so nobody gets credit for the first three quarters of the scale. The interval that measures training runs from 77.5 to the label ceiling at 87.0, and the relations-only student lands at 83.7 to 84.9 across seeds: about two thirds of everything label training could add, from a signal that never contained a label, an answer, or even a class.

Before celebrating, though, notice who the judge was. A linear probe is allowed to re-weight and shear the space however it likes to find the classes: it will happily rescue features that are merely present, recoverable along some direction, whether or not the space is actually arranged well. Maybe that is all relations buy, raw material a probe can salvage, not a geometry worth the name. If distillation truly transfers a geometry, the result should survive a ruler that is not allowed to fix anything.

The probe that isn’t allowed to cheat#

Nearest centroid is that ruler. Freeze the embedding, average each class’s embeddings into a single centroid, and classify every test image by whichever centroid sits closest. No learned weights, no re-weighting, no shear: either the classes sit in tight, well-placed clumps or the probe fails. It is the geometry-native measurement, and this time the expectation is lopsided. The label-trained student learned from the answer key itself; the relational student was never even told that classes exist. Surely the label student wins.

The relational student scores 81.2%, above the label-trained student’s 80.0% and essentially at the teacher’s own 81.7%. The student that was only ever told how pairs relate arranges its classes better than the student that was told the answers. That is the sentence “distillation transfers a geometry” turned into a measurement.

The inherited geometry even fails like the teacher. Rank every model’s confusions on the test set: the relational student’s confusion pattern correlates with the teacher’s at 0.957 (the random control manages 0.712). The teacher’s centroid probe sends stray Shirts mostly to T-shirt, Pullover, and Coat, at rates 0.26, 0.22, 0.15; the label-free student’s rates are 0.24, 0.23, 0.17. It did not just learn where the classes are. It learned which mistakes are natural, which is exactly the “better mistakes” structure the previous post located in the kernel.

Which spectrum does the student grow into?#

There is one more thing to check, and it is the one that makes the whole arc click shut. If a latent space is the spectral embedding of a class kernel, then a student trained on a kernel should not just probe well. Its latent geometry should become some specific kernel’s, eigenvalue by eigenvalue. But whose? Two candidates sit in this experiment, and they are not the same object: the teacher’s own latent geometry, the private arrangement it actually thinks in, and $S(T{=}4)$, the version of it that went through the softmax and out onto the wire. Watch epoch by epoch, from the real checkpoints:

Before the student’s number, settle what you expect. If anything in this experiment should agree with $S(4)$, it is the teacher: the kernel is a readout of the teacher’s own mind, computed from nothing but its outputs, so the teacher’s latent geometry should match it best and the student, a copy of a copy, should trail somewhere behind. Measured, the order flips. By the last epoch the student’s class-mean Gram agrees with $S(4)$ at CKA 0.976; the teacher’s own latent Gram agrees with it at only 0.854.

The resolution takes one turn: the softmax reshaped the teacher’s private geometry on the way out (the temperature dial again), so $S(4)$ is not the teacher’s mind but the teacher’s message, and the student inherited the message, not the mind. The copy is more faithful to the wire than the author is. Its spectrum sits at L1 distance 0.24 from the transferred kernel’s but 0.47 from the teacher’s latent, while the KD and label students, which fit answers rather than relations, stay close to the teacher-latent shape instead (L1 0.06 and 0.03). Nothing in the relational loss mentions eigenvalues. The student converges onto the spectral embedding of the kernel it was fed because, as the previous post argued, that embedding is what “a latent space matching a similarity” means. Distillation is that theorem running as a training loop.

What this does not say#

The claim “distillation is kernel transfer” earns its caveats:

• The relational channel is thin but not free. The student still needs the teacher’s outputs on the same inputs; the kernel arrives example by example, so the transfer set matters, exactly as Hinton’s held-out-3 experiment already showed. And our classic-KD baseline (85.8%) did not beat the label student (87.0%) here; on a small, clean dataset with abundant labels there is little for soft targets to add, and whether KD helps top-line accuracy is famously regime-dependent (Stanton et al., 2021). The point of this post is not that distillation beats labels. It is what distillation moves when it moves anything.
• The probes use labels. The student never sees them, but the measurement does; “label-free” describes the training signal, not the evaluation.
• The floor is high and must be named. Random features carry a lot of Fashion-MNIST by themselves (77.5% linear, 62.0% centroid). All fractions above are computed against that floor, not against chance, or the result would look more magical than it is.
• One dataset, one small architecture pair, three seeds. The numbers are stable across the seeds we ran (83.7, 84.5, 84.9 linear), but this is a measurement on Fashion-MNIST, not a law of nature.
• Relational distillation is not new as a method; matching pairwise structure is the RKD / similarity-preserving line of work (Park et al., 2019; Tung & Mori, 2019). What is different here is the question: those papers add relations to labels to climb a leaderboard, while this run removes everything except relations to isolate what they alone carry.

The picture, completed#

Three posts, one object. The Welch-bound post asked how class codes should pack, and Latent on the Spectrum answered that a latent space is the spectral embedding of a class-similarity kernel, with the packing as its structureless special case. What was missing was motion: kernels were things a designer chose. This post watched one travel. A trained teacher serializes its kernel through its softened outputs, one example at a time; a student that matches nothing but those pairwise relations reconstructs the embedding on its side of the wire, closes about two thirds of the random-to-labels gap, out-organizes the label-trained student on the geometry-native probe, and grows the transferred kernel’s spectrum, eigenvalue by eigenvalue.

Distillation works because the teacher’s outputs define a class-similarity kernel, and a kernel is a geometry a student can be trained into. The labels tell a student where each example goes; the kernel tells it how the world hangs together. In this run, most of what label training builds could be rebuilt from the hanging-together alone. The teacher was never handing over an answer key. It was handing over the shape of the space the answers live in.

And the temperature dial you dragged above is, in the end, the whole practical craft of distillation: choosing which geometry to put on the wire.

@misc{bouhsine2026distillationiskerneltransfer,
  author       = {Bouhsine, Taha},
  title        = {Distillation Is a Geometry, Not an Answer Key},
  year         = {2026},
  month        = {jul},
  howpublished = {\url{https://tahabouhsine.com/blog/distillation-is-kernel-transfer/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}