What a Finite Kernel Buys an MLP

Every multilayer perceptron is built from one primitive: a linear map followed by a pointwise nonlinearity, $h = \sigma(w^\top x + b)$. The activation $\sigma$ is the part nobody can quite justify. We reach for ReLU because it trains, GELU because it trains a little better, and we tell ourselves the choice is a detail. A pointwise activation is a patch over a missing object, and everything that makes an MLP opaque follows from the patch: no geometry, no centers, no attribution, no capacity you can name.

This post asks a single question. What if the primitive were not a linear map plus an activation, but a finite bank of explicit positive-definite kernel sections, each comparing the input with a learned center? The nonlinearity and feature-space geometry would come from the chosen kernel. Locality, normalized attribution, and generalization guarantees require additional conditions that the post will state explicitly.

It answers the constructive half of why activations are bad for geometry: if a pointwise activation damages the representation, replace the primitive with a declared similarity to a learned point.

First, the kernel you do not mean#

There is already a fashionable way a kernel shows up in a neural network, and it is the wrong one for this story. Take a network, send its width to infinity, and in the lazy regime it linearizes around initialization into a fixed kernel, the Neural Tangent Kernel (Jacot et al., 2018). The NTK is real and useful, but look at what it is: emergent (you do not write it down, it falls out of a limit), infinite-dimensional, fixed at initialization, and data-blind. It describes a network that has, in a precise sense (Chizat et al., 2019), stopped learning features, the parameters barely move, and the function is a kernel regressor against a kernel nobody chose.

That is the opposite of what I want. Here the bank is finite (the layer evaluates a declared number of sections), the kernel is explicit (a design choice, not an asymptotic limit), and the centers are learnable. The full Yat kernel is not finite-rank because its inverse-multiquadric factor has an infinite-dimensional feature map. At each frozen parameter value the layer is a finite expansion in a known kernel family; moving the centers makes training a nonconvex feature-learning problem rather than fixed-kernel regression.

The unit#

For an input $x\in\mathbb{R}^d$ and a learned center $W_u\in\mathbb{R}^d$, with two non-negative scalars $b\ge 0$ and $\varepsilon>0$, hidden unit $u$ computes

There is no $\sigma$. The numerator is a squared alignment (how much $x$ points along $W_u$); the denominator is an inverse-multiquadric distance gate (how close $x$ sits to $W_u$). On the unit sphere its response is a function of $S=W_u^\top x$. The center $S=1$ is the global peak, but for $b<1$ the square also creates a smaller antipodal lobe after the response crosses zero at $S=-b$. The kernel unit therefore carries a point and a direction-sensitive symmetry, not a generic one-bump radial profile. The contrastive geometry audit works through that boundary.

Two properties make this a kernel and not just a clever activation. It is pointwise non-negative, because the numerator is a square and the denominator is positive. It is also positive semidefinite for $b\ge0$ and $\varepsilon>0$: $(w^\top x+b)^2$ is a degree-two polynomial kernel, $1/(\lVert x-w\rVert^2+\varepsilon)$ is an inverse-multiquadric kernel, and their product is PSD by the Schur product theorem. Those facts are independent. PSD supplies an RKHS; pointwise non-negativity permits normalized convex weights. Universality is stronger and needs the constant term: the paper proves it for the admissible $b>0$, $\varepsilon>0$ regime. At $b=0$ the kernel is even under $x\mapsto-x$ in its alignment factor and cannot approximate arbitrary continuous targets on a symmetric domain.

Here is the whole thing learning. A layer of these units, one prototype each, no activation function, trained by gradient descent on a 2-D task. Watch the prototypes migrate onto the data and the decision field form. At every frozen parameter value the predictor is a finite expansion in sections of the same kernel family. The optimization itself is not fixed-kernel learning, because moving the centers changes the finite span being searched. Fixed-RKHS representer theorems apply only after the centers and kernel hyperparameters are held fixed.

The rest of this post is the catalogue of what that move buys you.

Feature 1, Lazy loading: the layer evaluates only what is near#

Because the response is usually largest near the center, a thresholded approximation can concentrate work on nearby prototypes. On normalized inputs the unit reduces to $k^\circ(S)=(S+b)^2/(\varepsilon+2-2S)$. It peaks at $S=1$, but the antipodal lobe for $b<1$ means a nearest-neighbor index alone is not always enough.

The exact layer is dense: every prototype returns a positive value. Retrieval becomes valid only after choosing an approximation tolerance and measuring the omitted mass or prediction error. That still creates a useful possibility, especially when responses are concentrated, but sparsity is an algorithmic approximation to certify rather than a theorem of the formula.

Feature 2, Explainability is not bolted on, it is the kernel theory#

A Mercer kernel comes with a reproducing kernel Hilbert space, and this learned-center architecture exposes several useful objects directly. The exact claims depend on the readout and on whether the kernel is held fixed; they do not transfer verbatim from normalized attention.

• A named center. Each unit’s output is a similarity to $W_u$, a point in input space you can visualize, decode, or compare. “What does this neuron detect?” has a literal answer: whatever lives at $W_u$. Ask a ReLU neuron the same question and it can only hand you back its half-space.
• Exact arithmetic attribution. A linear readout is a weighted sum of the displayed kernel evaluations, so each term’s contribution is directly computable. This identity does not require the representer theorem. That theorem becomes relevant for a fixed kernel and a norm-regularized empirical objective, where it says an optimizer can be chosen in the span of the training sections. When the readout uses non-negative normalized weights, the contributions form a convex partition.
• A section norm you can compute. The squared RKHS norm of the section $k(W_u,\cdot)$ is $k(W_u,W_u)=(\lVert W_u\rVert^2+b)^2/\varepsilon$. It measures the size of that section in the fixed RKHS; it is not by itself a confidence or generalization certificate for the trained network.
• A geometry on inputs. The kernel induces a pseudometric. It becomes a metric only if distinct inputs have distinct feature images, a property that must be checked for the domain in use.

The feature-space geometry and section norms come from the fixed kernel. Named centers and arithmetic attribution come from the finite learned-center parameterization and linear readout. Normalized percentage contributions exist only if the architecture explicitly normalizes nonnegative scores.

Feature 3, The geometry survives the layer#

A pointwise activation damages the geometry of the representation in ways that are now well documented: ReLU zeros coordinates and collapses rank, saturating activations flatten the pullback metric, and in high dimension these are the rule, not the exception, the full argument is activations are bad for geometry. The damage is structural: a function applied coordinate-wise cannot preserve angles and distances, because it does not know what the coordinates mean together.

The kernel unit does not apply a function to the representation; it measures the representation against centers. Its level sets are distance contours around $W_u$, and the geometry the layer exposes is the kernel’s induced geometry, informative by construction rather than degenerate by accident. You are not asking a nonlinearity to be selective without warping space, which is the impossible trade the activation is caught in. You are reading space with a ruler.

Feature 4, No activation lottery, but still stationary points#

The ReLU-versus-GELU-versus-SiLU question disappears, because the nonlinearity is now the chosen kernel family. The hard ReLU half-space disappears with it, but zero gradients do not disappear. A unit can stall when its readout coefficient is zero, when the numerator vanishes on the samples driving its update, or at an ordinary stationary point of the loss. The useful distinction is narrower: the kernel has no entire input half-space on which its response and center gradient are forced to vanish for architectural reasons.

Feature 5, A capacity functional, so regularization has a target#

For a fixed kernel with diagonal bound $k(x,x)\le\kappa^2$, an RKHS ball $\{f:\lVert f\rVert_{\mathcal H}\le B\}$ has empirical Rademacher complexity at most $B\kappa/\sqrt n$. For normalized inputs, $\kappa=(1+b)/\sqrt\varepsilon$ is one available bound for this kernel. Turning that fact into a guarantee for a network with trained centers, trained $b$, trained $\varepsilon$, and multiple layers needs a declared norm constraint and a separate analysis of the parameter-dependent class. The section norm therefore gives regularization a meaningful target, not an automatic generalization theorem.

Feature 6, The “finite” part: a feature map you can write down#

This is where the word finite needs care. The implemented network has finitely many sections, and the polynomial numerator has an exact finite-dimensional feature map of size $O(d^2)$. The full product kernel is not finite-rank because the inverse-multiquadric factor is not. Approximating that factor with random features or quadrature creates a second model whose Gram error, predictor error, and runtime must be measured separately. The attention companion performs one such finite approximation; it does not make the exact Yat RKHS finite-dimensional.

This is the kernel trick made spatial. The exact degree-2 feature map $\phi(x) = [x_1, x_2, x_1^2, x_2^2, x_1x_2, 1]$ turns a curved boundary into a flat one: a hyperplane in feature space is a conic in input space. Below, the flat separator is fitted live (a least-squares solve in feature space), and the data is lifted into 3-D by its quadratic score. Raise the lift and the classes rise apart until a single flat plane slices between them; look at the floor and that same flat plane is a circle (rings) or a hyperbola (XOR). The spiral is the counter-example: degree two is not enough, so it never fully separates.

Feature 7, Out-of-distribution inputs get a smaller answer, not a confident one#

Drive a ReLU MLP far off its training distribution and its logits can grow without bound along an active linear region. A local kernel exposes a different channel: the unnormalized response can shrink when an input is far from every prototype. Normalization deletes that evidence by forcing the weights to sum to one, and a softmax can still turn small or shifted scores into confidence. The abstention signal is therefore the field magnitude or maximum raw kernel response, measured and calibrated separately from the class probabilities.

Feature 8, One primitive for the whole network#

Attention has normalized weighted-average arithmetic over tokens. A kernel FFN has a finite expansion over learned centers. They can use the same base compatibility function, but their index sets, normalization, signed readouts, and symmetry conditions differ. The shared benefit is explicit pairwise scoring; there is no automatic single end-to-end RKHS, attribution guarantee, or capacity functional for the composed network.

What the learned objects look like#

On Fashion-MNIST, the centers live in pixel coordinates, so they can be rendered directly. That visibility is architectural: a center has an address in input space. Legibility is empirical. Example-seeded centers remain garment-like during training; noise-seeded centers can reach similar accuracy while remaining visually noisy.

With a linear readout, a logit is an exact finite sum,

Each displayed term is genuine arithmetic in that layer. It is not automatically a percentage share, a feature attribution to raw covariates, or a proof about the whole composed network. Those stronger readings require a normalization or a separate intervention.

The same named-center structure supports interventions. In the painting-arithmetic experiment, units were tagged by the operator glyph nearest the operator part of their centers. Zeroing the rows tagged as multiplication damaged multiplication much more than the other operations, whereas random rows of the same size caused diffuse damage. That is evidence from one architecture and task, not a theorem that nearest-center labels are universally causal.

What it costs#

This is not free. Whether the Yat MLP is a drop-in win over the transformer FFN at scale is an empirical question, the same boundary the attention piece drew for attention. Three real costs and caveats:

• Distances are not quite a matmul. $\lVert x - W_u\rVert^2 = \lVert x\rVert^2 + \lVert W_u\rVert^2 - 2 W_u^\top x$ reduces to a matmul plus two norm vectors, so the asymptotics match a linear layer, but the constant and the memory traffic are higher than a single GEMM.
• Composition changes the object. Any finite hidden representation induces a new PSD kernel by an inner product on its outputs, so Yat layers can be composed. What does not survive automatically is the original RKHS, its universality statement, or the input-space meaning of later centers. Each composed layer needs its own certificate and diagnostic.
• The scalars and centers need care. $b$ and $\varepsilon$ are admissibility-constrained (parameterize them through a softplus so they stay non-negative), and prototype initialization matters more than weight initialization in a ReLU net, because a center is a location, not just a scale.

The activation was always the missing kernel#

Strip the construction down and the claim is small. Replacing an affine-activation unit with a finite bank of input-space kernel sections makes the references explicit and gives the fixed base kernel a feature-space geometry. With a linear readout, each section’s signed arithmetic contribution is exact. Locality depends on the chosen kernel and domain, percentage attribution requires normalization, and fixed-RKHS capacity bounds do not automatically cover learned centers or a deep composition. Those boundaries leave a useful architecture without turning it into a universal explanation theorem.

The Yat kernel and its universality are from Bouhsine (2026); prototype networks from Chen et al. (2019); the geometric cost of activations in activations are bad for geometry; and the RKHS reading of attention in attention is explainable because it is a kernel. The Neural Tangent Kernel is Jacot et al. (2018) and the lazy regime Chizat et al. (2019); random features Rahimi & Recht (2007) and Choromanski et al. (2021). The runnable companion now includes both the 2-D construction and the Fashion-MNIST prototype audit: The Yat-Kernel MLP in JAX/Flax NNX.