Your Neuron Is a Direction. It Should Be a Picture.

Ask a trained network to show you one thing it knows, one thing you could point at, and it cannot. Not because the knowledge is missing, but because of what a neuron is. A neuron is one number it computes about its input, and for most of the history of the field that number has been

a projection of the input $x$ onto a weight vector $w_u$, then a squash. The weight is a direction, and that is the quiet problem we have stopped noticing: a direction is not a thing.

Ask what $h_u$ detects and there is no input to point at. The value depends on $x$ only through the scalar $w_u^{\top}x$, so the unit responds identically to every input on the hyperplane $w_u^{\top}x = \text{const}$ and grows without bound as you push along $w_u$. Its most-exciting input under a norm budget is just the weight itself, $\arg\max_{\lVert x\rVert=1} w_u^{\top}x = w_u/\lVert w_u\rVert$, which has no reason to look like anything. That is why reading a neuron needs an apparatus: feature visualization, probing datasets, saliency, sparse autoencoders, machinery built to manufacture a referent the unit never carried.

Change one thing. Replace the projection with a kernel against a point:

This is the Yat unit from the last post: large when $x$ is both aligned with $w_u$ and near it, with a single peak at $x = w_u$. So $w_u$ stops being a direction and becomes a prototype, a location in the same space as the data. Its most-exciting input is no longer a faceless direction; it is the prototype itself. A direction tells you which way. A prototype tells you which thing.

The gap between those two formulas is the entire post, and it has three consequences, each of which you can play with below.

1. The neuron becomes a picture. If $w_u \in \mathbb{R}^{784}$ lives where $28\times28$ images live, then reshaping it is an image. You read the unit by looking at it.
2. Training stops being where the story starts. A prototype is a labeled example, and that raises a question the middle of this post answers: how much can the network already do before the first gradient step?
3. Every decision is legible. Prediction, neuron, class, mistake, boundary: all of it is “this looks like that,” and each is readable on its own.

For something to point at, I trained a one-hidden-layer Yat-MLP on Fashion-MNIST, $784 \to 50 \to 10$, prototypes seeded from training images, no convolutions. That model, the one every interactive panel below runs live, reached 82.6% against a same-shaped ReLU MLP’s 85.9%; its weights and run record ship with this post in public/yat-fmnist/, and scripts/yat_fmnist_export.py re-evaluates the shipped weights at 82.5% on the full test set. Three points is the price. Here is what it buys.

Every neuron is a picture#

The two networks are nearly twins: same shape, same data, three points apart in accuracy. Put their hidden units side by side and they could not look less alike. On the left, the 50 Yat prototypes, each reshaped to 28x28 and bordered by the class it votes for: shirts, trousers, sneakers, bags, boots, a wall of templates the network compares every input against. On the right, the ReLU MLP’s first-layer weights, the nearest thing it has to a unit: noise. Why does swapping a projection for a kernel turn static into pictures? Look first, then we will say why.

Why is the left side legible? The route to the answer passes through a fact that should make it seem impossible. The Yat unit is a positive-definite kernel, $\phi_u(x) = k(w_u, x)$, so it carries a reproducing-kernel Hilbert space $\mathcal{H}$: a feature map $\Phi : \mathbb{R}^{784} \to \mathcal{H}$ with

and the network’s function lives in $\mathcal{H}$, which is large and, for this kernel, genuinely infinite-dimensional (Bouhsine, 2026, proves it universal). An infinite-dimensional function, stored as fifty pictures? That is the tension the representer theorem resolves, and it resolves it exactly: you never store anything in $\mathcal{H}$. The function is indexed entirely by its centers $w_u$, and those live in the input space $\mathbb{R}^{784}$, the very space the data lives in. The computation happens in the Hilbert space; the things you keep and look at stay in image space. $x$ and $w_u$ are the same kind of object, so a prototype is just another image.

A linear neuron’s weight is in $\mathbb{R}^{784}$ too, numerically, but it is used as a functional, $x \mapsto \langle w, x\rangle$: an element of the dual space, blind to everything orthogonal to $w$. It points along a direction; it does not sit at a location. The kernel is exactly the step that turns the weight from a co-vector into a point, $\langle w, \cdot\rangle \rightsquigarrow k(w, \cdot)$, and a point in image space is an image. That one substitution is the difference between the two galleries.

That is the claim made visible. And it points at something stronger, which is the real reason any of this works.

You can build it by hand, with no training#

Here is where a nice way to draw weights turns into something stranger. If a neuron is a labeled picture, then a classifier is just a list of pictures, and a list is something you can write by hand, no gradient descent required. Take a handful of training images per class as your prototypes and wire the readout to one-hot, $A_{uc} = \mathbb{1}[\text{class}(u) = c]$, so each prototype votes for its own class. The prediction is then just a tally of resemblances,

a working Yat-MLP that has seen exactly zero gradient steps. It labels an input by which prototype-pictures it most resembles. This looks like that, with nothing trained.

Slide the prototypes up below and watch the test accuracy climb on its own. With twenty k-means centroids per class, summaries computed from the training images rather than anything learned, and a one-hot readout, the sum-vote reaches 68.5% on Fashion-MNIST, and reading each class by its single best-matching prototype reaches 79.4%, none of it trained (scripts/yat_fmnist_export.py prints both).

This reframes the whole thing. The network is not a black box that training fills with meaning. It is case-based reasoning, and it already classifies the moment its prototypes are real examples. Training does not create the ability; it sharpens it, nudging the prototypes and learning a smarter-than-one-hot readout, carrying that hand-built score up to the 82.6% the trained model reached above. The JAX companion builds and trains exactly this.

If the construction works this well cold, what happens when training starts from it instead of from noise? The hand-built model becomes a warm start, and it pays off exactly where black boxes struggle. With only 40 training images per class, placed k-means prototypes score 65.9% before a single gradient step, and fine-tuning from there reaches 74.9%, against 70.1% for the same network trained from a random start. Because you choose the prototypes, you can also choose to represent the classes you care about: on a split where Pullover and Sneaker are starved to 30 training images each, warm-starting with placed centroids lifts their recall from 12% to 33% and overall accuracy from 70.0% to 75.9% (mean over five runs; scripts/yat_fmnist_export.py again). Gradient descent does not have to discover that the rare classes exist; the readout already tells it, and it only has to refine the boundary. White-box construction is not only an explanation after the fact, it is a way to put your prior knowledge into the weights before training starts.

And because the network reasons by resemblance, every part of it is readable by looking. The rest of the post is that: its predictions, its neurons, its classes, its mistakes, and the boundaries between the things it knows.

The network explains itself#

A legible unit is worth little if the prediction built from it is still a black box. The test of interpretability is not the weights at rest; it is the moment the model makes a call. So put the question to it directly: why this class, for this input? The answer is already inside the forward pass, because the output is a vote and you can read the ballots. Normalize the activations into weights $a_u(x) = \phi_u(x) / \sum_v \phi_v(x)$, with $\sum_u a_u(x) = 1$, and the prediction is a convex combination of the prototypes’ votes: this input is 40% that shirt, 25% that pullover, and the class is whoever the matched pictures elect. There is no separate explainer; the bars are the computation.

Switch the input to MNIST digits, which this model has never seen, and the explanation collapses: every $\phi_u(x)$ is small, the strongest match drops below a threshold, $\max_u \phi_u(x) < \tau$, and the network says “I don’t recognize this” rather than confidently calling a 7 a sandal. For the shipped model $\tau = 47.59$, the value recorded with its weights in public/yat-fmnist/; it is easy to place because the two distributions barely overlap, a median max-match of 231 on Fashion-MNIST test images against 30 on MNIST digits (scripts/yat_fmnist_export.py recomputes all three). A ReLU MLP has no such floor; off its training distribution it extrapolates a confident answer for anything. Knowing when it does not know is just the kernel being local.

The same picture reads the other way. Instead of “what is this input,” ask “what is this neuron”: click a prototype and you get the inputs it fires hardest on, plus the class it has become a detector for. In a ReLU network this needs a feature-visualization optimization or a probe dataset; here the answer is two pictures side by side, the prototype and the things that look like it.

And once more, in reverse: a class. The readout is linear, so a class is a weighted committee of prototypes, $\arg\max_c \sum_u A_{uc}\,\phi_u(x)$, and “what is a Sneaker” is answered by the few sneaker pictures with the largest readout weight for it. There is no class vector to decode; the class is a short list of examples.

Its geometry is a map you can walk#

One prototype is a picture; fifty of them are a map. Because they are points in pixel space, they have distances and neighbours, so the whole network’s sense of “kinds of clothing” can be laid out flat and walked. Sneakers fall beside ankle boots, shirts beside pullovers and coats, trousers and bags off in their own corners. The internal organization is not buried in a weight tensor; it is a chart you read with your eyes.

And the decision boundary, usually an abstraction, becomes a place you can stand. Blend one item into another and the prediction flips exactly when the nearest prototype changes from one picture to the next. The boundary is not a mysterious surface in 784 dimensions; it is the handover from one exemplar to another, and you can slide right through it.

Reading the mistakes#

When a network is wrong, it usually leaves you guessing why. This one hands you the culprit. Every misclassified image carries its reason: the prototype the kernel fired on hardest. Click any mistake and you see the input beside the look-alike prototype of the wrong class. It is a coat, but it matched a pullover prototype, so the model said pullover. That is not a heatmap hinting at “important pixels”; it is the actual evidence the decision rested on, which means you can act on it, by adding prototypes where the classes collide or relabelling the confusable ones.

Where the pictures come from#

So does the kernel hand you the pictures for free? Here is the experiment that says no, and it is the most instructive one in the post. Train the same network with the prototypes initialized from noise instead of examples and it classifies just as well, while the prototypes stay noise, all the way to convergence. Everything you have been scrolling through was decided before training started: legibility is chosen at initialization, and training never revisits the choice.

That splits the credit precisely. The design choice, a unit that measures the input against a center in input space rather than projecting it onto a direction, is what makes a picture possible: a center is a point in the same space as the data, so on images it is an image, on audio it would be a sound, on tabular data a named exemplar, and the interpretability lives wherever the data is interpretable. But the kernel does not hand you the picture. Legibility comes from the prototypes living where the data lives, which a cheap data initialization seeds and the kernel’s locality preserves. The kernel buys you a unit that can be a picture; anchoring it to the data, at init, is what makes it one.

This is the “this looks like that” idea from prototype networks (Chen et al., 2019), and the broader case for models interpretable by construction rather than explained after the fact (Rudin, 2019). What the kernel adds is that you do not need a special architecture to get there: an ordinary MLP, with the activation replaced by a kernel and the prototypes seeded from data, reads itself.

Where it goes#

Strip it down and the picture is simple. A neuron that stores a thing instead of a direction turns the network into a kernel of examples: it classifies by resemblance, so it works before you train it; it shows its reasoning in pictures when it is right and when it is wrong; and it goes quiet when it is looking at nothing it has seen. This one is a teaching setup, a single Yat layer on raw pixels, and it scales the obvious ways, the kernel on learned features (the ProtoPNet recipe), a clustering term to sharpen the prototypes, more width to close the last points to ReLU, each of which strengthens the same object rather than patching it. The last post said the opacity of an MLP was the price of giving up the kernel. This is what you get back when you keep it: a network you can build by hand, read by looking, and that answers which thing, not just which way.

The prototype-network idea (“this looks like that”) is from Chen et al. (2019); the case for interpretable-by-design models from Rudin (2019); Fashion-MNIST from Xiao et al. (2017); the Yat kernel from Bouhsine (2026). The conceptual setup is What a Finite Kernel Buys an MLP.