Where Does a Weight Live?

A neuron writes down $w^\top x$, and the two arrays may even have the same coordinates. Their roles are still different. The input is a data point; the weight defines a linear functional, or equivalently a direction after an inner product identifies vectors with functionals.

Your input $x$ is a point in a declared data domain: an image, a location in a cloud, a thing with domain-specific distances. The weight $w$ is used as an arrow or hyperplane normal: “project onto me and read off the result.” Although $w$ may be stored in the same Euclidean coordinate space, the model does not declare it to be a sample or a prototype. Asking which images it is near therefore has no built-in semantic meaning.

So: what kind of object is a weight, and when can it be tied back to examples? A reproducing kernel Hilbert space gives a precise answer: inputs map to feature vectors, functions live in the same Hilbert space, and regularized optimizers can often be expressed using kernel sections anchored at training points.

Two readings of the same handle#

What would an address for a weight even mean? You can feel the answer with your hands before any math: drag the weight and the test point below. In the first reading, the weight is what it is in an ordinary neuron: a direction. To use it on an input you drop the input’s shadow onto the arrow, and the number you get out is w · x. Notice that the weight is off on its own, anchored to the origin, pointing; it is nowhere near the data, and “how far is the weight from this point” has no answer.

Now switch the reading to “a place.” The same handle is suddenly a point in the data cloud, and the way you compare it to an input is not a shadow but a distance: how close does $x$ sit to $w$? That second reading is where everything else in this post comes from. Nothing about the data changed. We changed what kind of object the weight is allowed to be, from a direction you project onto into a location you can stand next to.

Why “separate spaces” is the real problem#

In the ordinary linear picture there are two roles. There is input space, where domain geometry is declared. And there is the dual role of w, which maps an input to a scalar and sets a hyperplane. In finite-dimensional Euclidean coordinates these roles can be represented by arrays of the same shape, but that coordinate identification does not turn a classifier normal into a data example.

You can numerically compute $\lVert w-x\rVert$ when the shapes match, but the model has not licensed that number as a distance between a learned rule and an image. The meaningful built-in operation is $w^\top x$. Interpreting w as a prototype requires an architectural assumption, not just matching dimensions.

That gap is why an ordinary weight is so hard to interpret. You want to ask of a learned w, “what does it look like, what is it near, what would I draw to show what it detects,” and the geometry refuses the question, because the weight does not live where the pictures do.

The wish, and the one map that grants it#

What would it take to relate a learned function to examples? First map each input into a common feature space. Then represent the function as a vector in that space, preferably one whose coefficients can be anchored to observed points.

A positive-definite kernel supplies such a feature map: $k(x,y)=\langle\phi(x),\phi(y)\rangle_{\mathcal H}$. Inputs appear as feature vectors $\phi(x)$, while an RKHS predictor is represented by some vector $f\in\mathcal H$ and evaluated by $f(x)=\langle f,\phi(x)\rangle_{\mathcal H}$. A general $f$ is not itself an input point. The special objects $k(q,\cdot)$ correspond to feature images $\phi(q)$, and finite sums of those sections anchor a predictor to named centers.

The cleanest way to feel the difference is to ask, of a fixed query point, “where is the input most similar to me?” and watch the answer under the two notions of similarity.

On an unbounded Euclidean domain, the dot product has no finite maximizer along the query’s ray. The radial kernel in the panel instead peaks at its center. Other kernels need not have that localization property, and on a bounded or normalized domain even the dot product has a maximizer. The architectural lesson is therefore specific: a centered local kernel makes the reference point explicit.

The punchline: the weight is built from the data#

So the predictor has an address in $\mathcal{H}$. But $\mathcal{H}$ may be infinite-dimensional, so what can an optimizer look like? For objectives whose data term depends only on the training evaluations and whose regularizer is a strictly increasing function of $\lVert f\rVert_{\mathcal H}$, the representer theorem says that an optimizer can be taken in the form

a weighted sum over the training points. The best weight is a combination of the data’s own kernel bumps. It does not float somewhere in an abstract space disconnected from your examples; it is anchored to them, built out of them, living in their span. The weight is the data, recombined.

Build one yourself. Each point below carries a coefficient αᵢ; drag a point up or down to push its coefficient positive or negative, and the decision surface (its boundary in black) bends to match. The whole surface is nothing but a sum of bumps sitting on the data. Press “fit the labels” and a real solve picks the αᵢ that separate the two classes, the representer-optimal weight, assembled entirely from the six points.

This is the moment the two spaces become one. There is no longer a weight space sitting apart from input space, holding directions. There is one space, holding the data and, in their span, the weight. To read the weight you do not project; you ask $f(x) = \langle f, \phi(x)\rangle$, which is just $\sum_i \alpha_i\, k(x_i, x)$, the data voting on the query by similarity. Every evaluation of the weight is a similarity query against the points it was built from. Even the question the old geometry refused, “which training images is the weight near?”, becomes a computation here: the kernel values $k(x_i, \cdot)$ against the training set, the Gram matrix, hold every such answer, and the companion post computes them in a few lines of JAX.

This is where the physical picture the rest of this series runs on comes from: the centers $x_i$ are masses, the coefficients $\alpha_i$ are their charges, and evaluating the predictor at $x$ reads the field they produce. We built one of those landscapes by hand in Your Network Is a List of Pictures. Under the objective conditions above, the RKHS result guarantees that some optimizer has this finite training-point expansion.

What the one space buys you#

Why go to the trouble? Because a weight that is allowed to be a place can solve problems that no direction can touch. Two nested rings, an inner disk inside an outer ring, are the standard example. No line separates them: rotate and slide a direction-weight as much as you like and you stay near a coin flip, because the classes are not side by side, they are one inside the other. Switch the weight to a place, drop a single point in the plane, classify by distance to it, and the problem falls apart the instant you put the point in the center.

That is one dividend of a suitable RKHS model. The predictor and the mapped inputs share a Hilbert space, and a representer theorem can anchor the fitted predictor to training examples. It does not make the predictor an input point or guarantee that RKHS distance has a visual interpretation.

The learned-center Yat architecture takes a further, separate step: it parameterizes each kernel section by a center in input coordinates. On images, that center can be rendered as a picture. This visibility comes from the parameterization; it is not a consequence of the representer theorem, and whether a learned center resembles the data manifold must be checked empirically. That is the narrower claim tested in What a Finite Kernel Buys an MLP.

The reproducing kernel Hilbert space is Aronszajn (1950); the representer theorem in its general form is Schölkopf, Herbrich and Smola (2001); the standard reference is Schölkopf and Smola (2002); the input-space-center kernel is Bouhsine (2026). The runnable companion builds the representer-theorem weight in JAX: Where Does a Weight Live, in JAX/Flax NNX.