What Attention Weights Can Explain

Open a transformer block and you find two layers sitting on equal footing. They share the residual stream, they consume comparable parameter budgets, and they were designed as a pair. Yet practitioners read one and not the other. Attention heads earn names for the algorithms they run: induction heads, copying heads, name-mover heads, positional heads. The position-wise MLP next door gets called “computation,” “feature synthesis,” or simply “the part we don’t yet understand.”

The disparity is so familiar that it is easy to mistake for a fact about visualization tooling: attention hands you a matrix you can plot, and the MLP does not. But the MLP has plottable matrices too, and plotting them has never produced an induction head. Something else separates the two layers, something the architecture diagram does not advertise. What does attention have that the MLP lacks?

The asymmetry that the architecture does not advertise#

The answer turns out to be older than the transformer, and it comes from classical statistics: attention has the algebra of a normalized compatibility smoother. It becomes a classical kernel smoother only when the compatibility function satisfies the corresponding kernel conditions.

Once one sees attention as a kernel smoother, the things practitioners do when they “explain a head” (reading off pairwise affinities $q_i \cdot k_j$, attributing portions of the output to specific tokens, reasoning about locality and retrieval) turn out to be the operations that kernel smoothers were designed to support a half-century before transformers existed. The MLP, by contrast, has no kernel and so admits none of these operations natively. Whatever interpretability one can extract from it has to be imposed externally, by training a separate decoder, projecting into a learned dictionary, or otherwise constructing the geometry that the layer itself does not carry.

Follow that thought all the way down and it becomes a claim about the whole disparity. Everything practitioners find “explainable” about attention is downstream of the single structural fact that attention has a kernel, and therefore a geometry, a normalized contribution mass, and (when one is willing to symmetrize) an RKHS in which to reason about it. A standard ReLU MLP has none of these, not because it is more powerful, but because it is not a kernel machine.

The mathematical observation itself is not new. That self-attention is a kernel smoother is due to Tsai et al. (2019), developed further by Song et al. (2021), Choromanski et al. (2021), Katharopoulos et al. (2020), and Han et al. (2022), mostly in service of designing efficient attention variants. What seems to me underdiscussed is what the reformulation explains: it is the reason attention is explainable in the first place.

Attention as a kernel smoother#

What was a statistician in 1964 supposed to do with a scatter of noisy observations and no model to fit? Nadaraya and Watson’s answer was almost embarrassingly direct: to predict at a point, let every observation vote, and weight each vote by how close that observation sits to the point you are asking about. The “how close” is a kernel, a bump centered on the query. Slide the query and the bump slides with it; observations under the bump dominate the vote, observations outside it barely count. The prediction is a locally weighted average that follows whatever the data does nearby, with no global model anywhere in sight.

Now hold that picture and write down the definition every reader knows. Given a token sequence with queries $Q$, keys $K$, and values $V$ obtained by learned linear projections, scaled dot-product attention returns:

or, written token by token,

And here is the smoother from the panel above, in symbols. Given observations $(x_i, y_i)$ and a query point $x$, the Nadaraya–Watson estimator is:

These two expressions have the same normalized weighted-average form.

The quotation marks around “kernel” matter. Queries and keys are produced by different maps, so the compatibility $c(x_i,x_j)=\exp(q_i^\top k_j/\sqrt d)$ is generally asymmetric and need not define a PSD Gram matrix on tokens. The layer is Nadaraya–Watson-like in its normalization algebra, but it does not inherit an RKHS until symmetry and PSD are established. The weighted-average reading survives; Mercer theory does not automatically follow it.

Three affordances kernel structure gives you#

What does any of that buy the person squinting at an attention map? Three things, and each one is a move practitioners already make without naming it.

1. The kernel is an explicit pairwise score. For every pair of tokens $(i, j)$ there is a single real number $q_i \cdot k_j$ that summarizes how relevant token $j$ is to token $i$ under this head. Visualizing a head means visualizing the matrix of $\alpha_{ij}$. Comparing two heads means comparing two such matrices. The algorithmic descriptions one finds in the mechanistic interpretability literature (“this head copies from the most recent occurrence of the current token,” “this head attends from each token to its syntactic head”) are statements about the structure of this matrix. The kernel supplies a geometry on tokens, and geometry is the kind of object humans can reason about.

2. The weights normalize. The softmax enforces $\sum_j \alpha_{ij} = 1$ with $\alpha_{ij} \ge 0$, so each output $y_i$ is a convex combination of value vectors. This is the source of every attribution-style statement one ever makes about attention. When we say a head “moved information from token $j$ to token $i$,” we mean $\alpha_{ij}$ was large. When we say a head “ignored token $j$,” we mean $\alpha_{ij}$ was small. These statements are coherent precisely because the weights are normalized contribution masses, not arbitrary activations.

A standard linear layer, where some output coordinate is a linear combination of input coordinates with weights that can be positive, negative, or large in magnitude, does not admit this reading at all. The weights are not a partition of unity, do not compose across layers in any attribution-respecting way, and need not even be of consistent sign for nearby inputs. The fact that attention does admit it is again a direct consequence of its being a kernel smoother. In the Nadaraya–Watson form the weights normalize for the same reason and serve the same role.

3. Kernels carry locality. Kernel smoothers, by construction, weight nearby points more heavily than distant ones, and the notion of “nearby” is whatever the kernel says it is. In attention, “nearby” is high $q_i \cdot k_j$, which the model is free to shape during training. The upshot is that attention behaves, by default, like a content-addressable nearest-neighbor retrieval over the sequence: $j^{*}_i = \operatorname*{arg\,max}_j q_i \cdot k_j$ in the sharp limit. This is how it gets used in many of the algorithms that mechanistic interpretability has uncovered. Induction heads retrieve previous occurrences of a token. Name-mover heads retrieve antecedents. Positional heads retrieve fixed offsets. None of these descriptions requires anything beyond the kernel-smoother view; they are different specializations of “look up the most similar previous token under this kernel.”

And, when you symmetrize, an RKHS#

A pairwise score, a contribution mass, a locality: is that the whole inheritance, or does the kernel go one level deeper? One more level exists, weaker but worth having. If one is willing to symmetrize the kernel, replacing the query/key asymmetry with a single symmetric, positive-definite kernel $K(z, z')$ acting on a shared representation (as several variants in the literature do), then attention lives inside a reproducing kernel Hilbert space

The Han et al. analysis makes this explicit: attention can be read as kernel density estimation, and KDE itself is a kernel regression problem in an RKHS.

The reason this matters for explanation, beyond the formal pleasure of having a Hilbert space, is that the RKHS view supplies two further objects that a generic layer does not have.

The first is a function-level norm. For a fixed symmetric PSD kernel, the function $f\in\mathcal H_K$ has a well-defined $\lVert f\rVert_{\mathcal H_K}$. Bounds derived from that norm still need a declared loss, norm radius, diagonal bound, and sampling model.

The second is a dictionary. The kernel sections $\{K(\cdot,k_j)\}$ span the finite function produced by one attention row, but they need not be linearly independent and therefore need not form a basis. The layer computes an expansion

with explicit coefficients. These are arithmetic contribution weights for that layer, not automatically causal explanations. If the section Gram matrix is singular, the same function can also admit more than one coefficient representation.

Even where the strict RKHS structure breaks (because the attention kernel is non-symmetric), the kernel-smoother view retains a geometry on tokens: a learned distance, an explicit similarity score. It is this geometry that makes attention legible. And since the other layer in the block sits in the same residual stream and was trained by the same descent, it is natural to expect the same reading to carry straight over.

Now run the same reading on the MLP#

The playbook that worked for attention has three steps: find the pairwise score, read off the contribution masses, use the kernel’s geometry to say what “similar” means to the layer. By every expectation of symmetry the same three steps should go through next door. A position-wise MLP is, by definition,

typically with $\sigma$ a ReLU or GELU.

Now look for step one. There is no kernel $K(x, x')$ in this expression, learned or otherwise. There is no similarity score one can point at. There is no normalized weighting of inputs into outputs. There is no notion of which directions in $\mathbb{R}^d$ the layer treats as “nearby.” There is no Hilbert space in which the function $\mathrm{MLP}$ lives with a controllable norm. The reading does not go through, not at step three or step two, but at step one: the object it starts from does not exist.

The pre-activations $(W_1 x + b_1)_i = w_i \cdot x + b_i$ are linear features of the input, but a linear feature is not a kernel: it scores in one direction only, $w_i \in \mathbb{R}^d$, and supplies no geometry on pairs $(x, x')$. The ReLU nonlinearity then composes with the second linear map $W_2$, which mixes neurons arbitrarily and erases any chance that an individual unit corresponds to a human-readable feature.

And you can watch the missing geometry directly, because it is a property of the primitive, not of anything training adds. Take two untrained layers, feed both the same random input pairs, and ask each layer’s own similarity score to predict how far apart the two outputs land:

The left cloud bends along one curve; the right cloud is a smear. A kernel layer’s outputs track the geometry its kernel defines before a single gradient step, so the score on the horizontal axis is a genuine explanation of the behavior on the vertical one. The ReLU layer offers no score with that property, and no amount of training will conjure one, because the parameterization has nowhere to keep it.

This offers one structural explanation for the now-standard observations about MLP neurons. Their decomposition is basis-dependent because the parameterization does not nominate a canonical semantic basis. Superposition is not the absence of all geometry, since any hidden representation induces similarities such as $h(x)^\top h(x')$. The narrower claim is that the ordinary MLP does not expose a normalized token-to-token contribution matrix or a fixed input kernel as part of its forward computation.

Attention, by contrast, does carry a geometry: the kernel. And so it exposes its features in that geometry, in the only basis the model already uses, namely the token basis. Both layers sit in the same stream with comparable capacity. Only one of them keeps a geometry while it computes.

But isn’t this just the price of universal approximation?#

The natural objection here is that universal approximation forces this state of affairs. A function class powerful enough to approximate arbitrary continuous functions cannot afford the rigidity of a kernel expansion.

The objection conflates the function class with the parameterization. Some classical kernels are universal: their RKHS is dense in $C(\mathcal X)$ for the declared compact domain. Characteristicness is a different property, injectivity of probability-measure embeddings, and does not replace the universality hypothesis in general. A finite-sample, norm-regularized optimizer for a fixed universal kernel admits a representer expansion

in kernel sections with closed-form norm $\|f\|^2_{\mathcal{H}_K}=\alpha^\top K\alpha$. The whole RKHS is the completion of such finite spans; not every member is itself a finite expansion.

The reason a transformer MLP is not such an object is not that the function class is too rich. It is that the architecture has chosen the cheapest possible primitive (an affine map followed by a pointwise nonlinearity) and accepted the loss of structure as the cost. This is a design decision, not a theorem.

Recent work makes the point constructively. The Yat kernel,

is a hidden-unit primitive that is PSD for $b\ge0$, $\varepsilon>0$. For $b>0$, its constant polynomial component gives a positive multiple of the inverse-multiquadric kernel in the PSD decomposition, supporting the paper’s universality and characteristicness result on its stated domain. The $b=0$ case does not inherit that argument. A finite layer is a learned-center expansion

whose squared fixed-kernel RKHS norm is $\alpha^\top K\alpha$ once its centers and hyperparameters are frozen. Optimizing those centers is a nonconvex search over changing finite spans, not the fixed-kernel representer problem.

I am not arguing here that Yat-style MLPs are a practical replacement for the transformer FFN. That is an empirical question and not the subject of this piece. The point is to close the explanatory loop. The opacity of standard MLPs is not the price of expressivity. It is the price of giving up the kernel.

What this does and does not claim#

Attention weights are not the same as explanations, and the literature contains a well-known back-and-forth on how far one can trust them. What the kernel reading provides is not faithfulness but affordances: the structural objects (pairwise scores, normalized contributions, a geometry, sometimes an RKHS) that any explanation has to be grounded in.

The MLP lacks those objects natively. Until it has them, every explanation of MLP behavior has to import its geometry from elsewhere, whether through a sparse dictionary, a probe, or a trained decoder. Until the kernel is restored, the work of explaining MLPs will continue to be the work of supplying, after the fact, the structure that attention has carried all along.

References inline. The kernel-smoother view of attention is due to Tsai et al., with related developments by Song et al., Choromanski et al., Katharopoulos et al., and Han et al.. The Yat kernel is from Bouhsine, 2026. The superposition framing is from Elhage et al., and the mechanistic interpretability descriptions are from the circuits thread and related work.

@misc{bouhsine2026attentionisakernel,
  author       = {Bouhsine, Taha},
  title        = {What Attention Weights Can Explain},
  year         = {2026},
  month        = {may},
  howpublished = {\url{https://tahabouhsine.com/blog/attention-is-a-kernel/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}

@article{bouhsine2026260503262,
  author        = {Bouhsine, T.},
  title         = {A Universal Reproducing Kernel Hilbert Space from Polynomial Alignment and IMQ Distance},
  year          = {2026},
  eprint        = {2605.03262},
  archivePrefix = {arXiv}
}