What Can a Weight Be?

A kernel gives a weight a home: a space where the weight is a point among the data, built from the data, as the last post argued. That settles where the weight lives. It leaves a sharper question wide open. Once it has a home, what is the weight actually allowed to be?

The question is not idle, because two kernels can both make the weight a point among the data and still disagree completely about its character. Pick one kernel and the weight can have a sharp corner, a kink, a sudden spike. Pick another and that corner is not merely discouraged, it is impossible: no weight in that space has one, ever. Nothing in the data changed, nothing in the picture from last time changed. The kernel, quietly, decided what kinds of function the weight is permitted to be. So not all of these homes are the same, and it is worth knowing what kind you just moved into.

A kernel is a price list for roughness#

What decides who gets in? Money, in effect. Picture every candidate weight arriving with a shopping list: it is assembled from a fixed catalog of basic shapes, and each shape has a posted price. Smooth, slow shapes are cheap; sharp, fast, wiggly ones are expensive, and how expensive depends on the kernel. The space admits exactly the weights that can afford their total bill. A kernel is, secretly, a price list for roughness.

That picture is not a metaphor bolted on afterward; it is the literal content of Mercer’s theorem. Every kernel splits into a set of modes $\phi_i$, fixed shapes, each with a non-negative weight $\lambda_i$:

The reproducing kernel Hilbert space built from $k$ is then nothing but the functions you can write in those modes, $f = \sum_i c_i \phi_i$, with a very particular notion of size:

That norm is the bill. To use mode $i$ with amplitude $c_i$ you pay $c_i^2/\lambda_i$. A mode with a large eigenvalue $\lambda_i$ is cheap; a mode with a tiny eigenvalue is ruinously expensive. The RKHS is exactly the set of functions whose total bill is finite, the functions you can afford. And since sharp features, corners and spikes, are built from the high-frequency modes, whether the weight can be rough comes down to one thing: how fast the eigenvalues decay. That single curve is the kernel’s whole personality.

Sobolev: the home with room for a corner#

So which kernels let the weight be sharp? A large family of them, the Matérn kernels of spatial statistics, the Laplace kernel $e^{-\lVert x-y\rVert}$, and the inverse-multiquadric, all have eigenvalues that decay like a polynomial in the frequency, $\lambda_k \sim (1 + \lVert\omega\rVert^2)^{-s}$. Slow decay means high frequencies are merely expensive, not banned.

That has an exact name. The RKHS of such a kernel is norm-equivalent to a Sobolev space $H^s$, the functions whose first $s$ derivatives are square-integrable:

For the Matérn kernel of smoothness $\nu$ in $d$ dimensions, $s = \nu + d/2$. The point of a Sobolev space is that it is the natural home of functions with a finite amount of smoothness. A function with a corner has a finite Sobolev norm for small enough $s$, so it sits comfortably inside. The weight is allowed to be sharp. This is the right-sized home: big enough to hold the rough functions real data needs, structured enough to still penalize nonsense.

The Gaussian: a home too small to bend#

Now the cautionary opposite, and it is the kernel almost everyone reaches for first. The Gaussian, or RBF, kernel $e^{-\lVert x-y\rVert^2 / 2\sigma^2}$ has eigenvalues that do not decay like a polynomial. They fall off like a Gaussian themselves, faster than any power of the frequency. So the high-frequency modes are not expensive, they are astronomically expensive, and the bill for any function with a sharp feature is not large but infinite.

The consequence is severe and not widely felt: the RKHS of the Gaussian kernel contains only analytic functions, the infinitely smooth ones, and it is a strict subset of every Sobolev space at once. A function with a corner is not in it. A bump that is exactly zero outside a small region is not in it. The Gaussian kernel does not just prefer smooth weights, it makes a non-smooth weight literally impossible to express. You feel this the moment you fit data with a real edge in it.

The sphere: the home of normalized data#

Both of those homes sit in flat space, but the data in this series does not. We L2-normalize every feature vector, which puts every point on a sphere, and on a sphere the natural kernels are the ones that depend only on the angle between two points, $k(x, y) = \kappa(\langle x, y\rangle)$. These are the zonal, or dot-product, kernels, and Schoenberg settled their structure in 1942: such a kernel is positive definite on the sphere exactly when it expands in Gegenbauer polynomials with non-negative coefficients.

What makes the sphere beautiful is that its modes are already famous. The eigenfunctions are the spherical harmonics $Y_{k,j}$, the standing-wave patterns of a sphere, graded by a degree $k$: degree zero is a constant, degree one is a single smooth swell, and the patterns ripple faster as $k$ grows. A zonal kernel assigns one price $b_k$ to an entire degree at once, and its RKHS is the functions on the sphere whose harmonic content it can afford:

Same skeleton as before, a price list over modes, but now the modes are the vibrations of a sphere and the smoothness of the weight is set by how fast the kernel lets the degrees fade.

The rest of the zoo, in one line each#

And the rest of the zoo? The same lens sorts every kernel you have met, one line each. A polynomial kernel keeps only finitely many modes alive, so its RKHS is finite-dimensional: the weight can only be a polynomial of bounded degree, and there is no room for anything finer. A bandlimited, or sinc, kernel sets its prices to a hard step, one below a cutoff frequency and zero above, so its RKHS is the bandlimited functions and the cutoff is an absolute wall. The Brownian-motion kernel $\min(s, t)$ sits at the opposite extreme, with such slow decay that its space contains genuinely jagged paths, the $H^1$ functions. In every case the story is the same sentence: the eigenvalue decay is the personality, and the personality is the only thing that matters.

Where the prototype lives#

So when this series puts a prototype in the space, which home does it move into? The Yat kernel is a squared alignment $(\langle x, W\rangle + b)^2$ times an inverse-multiquadric distance gate $1/(\lVert x - W\rVert^2 + \varepsilon)$, and the gate sets its smoothness: the numerator is only a polynomial factor, and multiplying by a polynomial cannot change the decay class the gate imposes on the spectrum (the paper carries the full spectral argument). It is tempting to call that gate Sobolev, but the eigenvalues say otherwise: the inverse-multiquadric’s decay is exponential, faster than the Matérn family’s polynomial $k^{-2s}$. So the Yat kernel does not live in a Sobolev space, and, strictly, a corner is not in its RKHS either; the bill just diverges slowly rather than fast.

What its home is instead is the more useful thing here. Its eigenvalues fall slower than a Gaussian’s, so its space is strictly larger, and it is universal for $\varepsilon > 0$: it can approximate any continuous target as closely as you like. On normalized inputs it is a zonal kernel of $\langle x, W\rangle$, so its weights are graded by the spherical harmonics. And the property that makes a prototype a picture is on a different axis from frequency smoothness altogether: the inverse-square law in the gate is a heavy tail in space, which buys locality, a bounded response, and a center that stays in input space.

That is the whole answer to what a weight can be. It can be exactly as rough as the kernel’s eigenvalues let it afford to be, on exactly the geometry the kernel is built around. Choosing a kernel is not choosing a similarity score; it is choosing a smoothness class and a shape of space, and signing your weight up to live there. The prototypes of this series are sharp, legible pictures not because their home is Sobolev-rough, but because it is local, universal, and on the sphere, the one place where a center can stay a picture you can point at.

Positive-definite kernels on the sphere are Schoenberg (1942); native spaces and the Sobolev correspondence are in Wendland (2004) and Stein (1999); the smallness of the Gaussian RKHS is in Steinwart and Christmann (2008); the polynomial-alignment and IMQ construction is Bouhsine (2026). The runnable companion computes these spectra and RKHS norms in JAX: What a Weight Can Be, in JAX/Flax NNX. The conceptual predecessor is Where Does a Weight Live?.