The Kernel Between the Roles

Two threads of this blog have been walking toward each other for months. One read attention as kernel machinery: a Nadaraya-Watson smoother whose exp(q·k) plays the kernel’s part without being one. The other built a program on a kernel that actually is one, the Yat kernel, and spent it on MLPs, editing, survival models, fixed points. The QK post closed on the meeting point and then refused to build it: the trap, it said, is that a Mercer kernel on tokens is symmetric, which deletes the query and key roles that make attention directional; the right move is to kernelize the compatibility between the roles, $s_{ij} = \kappa(f_Q(x_i), f_K(x_j))$, and that construction deserves a post of its own. This is that post.

What the exponential was hired to do#

Strip attention to its bookkeeping and one requirement stands out. The mechanism turns scores into a weighted average, so the weights must be nonnegative and sum to one. Bilinear scores $q \cdot k$ come out of a matrix multiply signed, positive here, negative there, useless as weights. The exponential is the repair: $e^x$ is positive whatever $x$ was, so wrap the scores and divide by the sum. That is the whole softmax, and it works, and everything distinctive about softmax attention follows from that one repair rather than from anything attention asked for. The scores gain a gauge: shift every logit by a constant and the weights do not move, so the absolute level of compatibility is deleted information, the same shift-invariance that deletes a kernel’s field magnitude in a classifier head. The implementation gains the max-subtraction trick, because $e^{30}$ overflows what floats can hold. And the geometry gains the property the QK post lingered on: $e^{q \cdot k}$ behaves like a kernel smoother operationally while being no Mercer kernel at all.

But suppose the score arrived positive on its own. A Mercer kernel with a nonnegative range needs no repair: weights are $\kappa_{ij} / \sum_j \kappa_{ij}$, done. No exponential, no gauge, no overflow trick, and the smoother is not an analogy anymore, it is the literal Nadaraya-Watson estimator that the first attention post could only read into the formula. The Yat kernel is exactly such an object (arXiv 2605.03262):

Positive semidefiniteness is a fact, not a hope, and the proof fits in a breath: $(q \cdot k + b)^2$ is the quadratic polynomial kernel, $1/(\lVert q - k \rVert^2 + \varepsilon)$ is the inverse multiquadric, and the Schur product theorem says the pointwise product of two kernels is a kernel. The bias deserves a sentence of respect before anything trains, because it is not a convenience constant. Expand the numerator: $(q \cdot k)^2 + 2b\,(q \cdot k) + b^2$, a quadratic, a linear, and a constant term in the alignment. Drop $b$ and only the even term survives, and the paper’s universality theorem fails with it: with $b > 0$ the product kernel’s reproducing-kernel Hilbert space is dense in the continuous functions, a universal kernel, while at $b = 0$ that density is lost. The bias is what buys the theorem. And $\varepsilon$ is the same softening the prototype posts used, capping the kernel’s peak at $(\lVert q \rVert^2 + b)^2 / \varepsilon$ when key meets query. Both scalars are learned, per head, through a softplus, initialized at $\log 2$: each head gets to choose its own floor of compatibility and its own softening, forty-eight parameters in total, and what they learn is itself telemetry. The full apparatus, RKHS, feature map, representer theorem, everything this series built its network program on, rides on top.

And the trap from the QK post is disarmed the way that post prescribed. The kernel is symmetric in its two arguments, but its arguments are not tokens: they are $f_Q(x_i)$ and $f_K(x_j)$, the learned query and key projections. Directionality lives in the roles, $f_Q \neq f_K$, exactly where it lived under softmax. Only the comparison between the roles changes.

What the kernel scores that a dot product cannot#

Before any training, the two compatibility functions disagree about geometry, and the disagreement is visible. A bilinear score rewards one thing: projection onto the query’s direction. It grows without bound along that ray and cannot tell a key sitting on the query from a key ten times farther out along the same line. The Yat kernel multiplies alignment by proximity: its numerator wants the directions to agree, its denominator wants the vectors near each other, and its peak sits at the query itself.

This is the prototype geometry this series built for neurons transplanted into attention: under a Yat score, a key is not a direction to project onto but a place, and a query attends hardest to keys that are near it in the projected space. Whether a language model wants that inductive bias is precisely what an existence-proof experiment is for.

What each normalizer forgets#

The second structural difference needs no training either. Softmax’s shift gauge means the row of weights is all that survives; the level of the scores, how compatible the best match actually was, is deleted by construction. The kernel’s normalization also produces weights summing to one, but the raw row sum, $m_i = \sum_j \kappa_{ij}$, is a real nonnegative number that survives as a separate channel: how much total compatibility this query found, before normalization spent it.

The mass channel is the attention version of the kernel field magnitude that a bounded network keeps beneath its softmax, the quantity the prototype network’s abstention story was built on. Structurally, it is free information. What it is worth in a transformer, the run below measures, and the measurement deserves to be reported exactly as it came out.

The run#

Two character-level GPTs on Shakespeare, identical in every respect except the score slot: 6 layers, 4 heads, model width 192, context 128. The softmax model computes $\mathrm{softmax}(q \cdot k / \sqrt{d})$ at 2,719,169 parameters; the Yat model computes $\kappa / \sum \kappa$ with its per-head learned $b$ and $\varepsilon$, 2,719,217 parameters, the difference being exactly those forty-eight scalars, and contains no softmax anywhere in its attention. Each side trains at its own swept learning rate, 3e-4 for softmax and 3e-3 for the kernel, per the fairness audit below. Three seeds each, same data order, 12,000 steps, best validation loss (scripts/yat_attention.py, bundles kgl_blog-yatattn-v1 and kgl_blog-yatattn-b-v1).

Softmax finishes at 1.4923 ± 0.0060 nats per character, the kernel at 1.5089 ± 0.0020, a 1.1 percent gap. The learning rates deserve their provenance, because this series has asked the fairness question before, in the survival-model trial: each side must train at its own best rate. A full sweep, both mechanisms at 3e-4, 1e-3, 3e-3, and 1e-2, found softmax’s optimum at 3e-4 and the kernel’s at 3e-3, the roughly tenfold ratio the Yat training experience predicts. The sweep’s shape is worth more than its argmaxes: softmax’s quality lives on a cliff, best in one narrow window, degraded at 1e-3, and at 1e-2 it diverges outright to 2.58 nats, an untrained model, while the kernel walks the entire 33-fold range without leaving a one percent band. An exponential amplifies whatever scale the gradients hand it, so softmax’s learning rate carries the job of keeping the logits in the survivable window; a ratio of polynomials normalizes whatever arrives. The kernel spends no hyperparameter budget on staying alive.

And the bias earns its theorem-fee in the ablation. Set $b = 0$, the non-universal special case, and three-seed quality slips to 1.5131 ± 0.0071, a loss at the boundary of run-to-run noise, but the score telemetry moves decisively: the ablation’s maximum trained score runs to 443,508 against the full kernel’s 61,221. There is also a routing option the ablation deletes outright: at $b = 0$ an orthogonal query-key pair scores exactly zero regardless of how near each other the two vectors sit. Universality is not an abstraction fee, it is these degrees of freedom.

And the forty-eight scalars use their freedom. Exported from the trained model, the biases have wandered from their $\log 2$ initialization across the range 0.22 to 3.40, every layer holding heads that raised their compatibility floor and heads that lowered it. The softenings split more dramatically: most heads drive $\varepsilon$ toward zero, sharpening their kernel’s peak toward a nearest-neighbor router, while two late-layer heads grow $\varepsilon$ to 6.4 and 12.7, spreading themselves toward global averagers. Per-head kernel scalars are not a numerical nicety; they are each head choosing what kind of statistician to be, and the choice is readable off two vectors of numbers in a way no $W_Q W_K^\top$ ever is.

What the sweep found instead is a difference worth more than the gap. Softmax’s quality lives on a cliff: best at 3e-4, already degraded at 1e-3, and at 1e-2 it diverges outright, 2.58 nats, an untrained model. The kernel walks the entire 33-fold range and never leaves the 1.50 to 1.52 band. At every learning rate except softmax’s own optimum, the two mechanisms tie or the kernel wins. The mechanism is the one this post already named: an exponential amplifies whatever scale the gradients hand it, so the logits must be kept in a narrow band and the learning rate does that job; a ratio of polynomials normalizes whatever scale arrives. The kernel spends no hyperparameter budget on staying alive. Per the standing frame of this whole program: the number is not a victory and is not trying to be one. The claim is the existence proof, the same one the solve post made for kernel machines and the Yat MLP posts made for networks: a transformer whose routing is a genuine Mercer-kernel smoother, with no exponential in the path, trains by plain gradient descent to within a hair of the mechanism that has monopolized the architecture since 2017, while keeping the kernel’s structure, the geometry, the mass channel, the theory, intact underneath.

And here is what the trained routing looks like, side by side on one Shakespeare window, from the telemetry run’s checkpoints:

Similar, but not the same, and the difference is measurable rather than aesthetic. Score each trained query row by its normalized entropy, zero for a row that stakes everything on one key, one for a row spread evenly, and the two mechanisms separate cleanly: the softmax model averages 0.48, with 22 percent of its rows sharpened nearly to a single key; the kernel model averages 0.73, with 0.2 percent of rows that sharp and 37 percent diffuse. This is not a training accident, it is the arithmetic of the two normalizers. An exponential turns score gaps into weight ratios that grow without limit, which is why softmax is rightly described as a soft argmax: give one key a few logits of advantage and it takes essentially everything. A ratio of polynomials cannot do that; for one key to take 99 percent of the row, its kernel value must exceed the sum of all others hundredfold on the raw scale, not the log scale. The kernel buys its geometry at the price of concentration, and that price is a plausible reading of the 1.4 percent: character-level language leans hard on copy-like heads that want to be nearly one-hot. It also names the obvious dial for closing the gap, raising the numerator’s power, $(q \cdot k)^{2m}$, which sharpens the kernel’s peak while every structural property above survives.

Two of our beliefs, measured#

An existence proof earns its keep by what it refuses to fudge, and this run falsified two things we expected to write.

The first was the word bounded. The Yat kernel is bounded for bounded inputs, and we expected the kernel scores to stay tame while softmax logits roamed. The opposite happened. The trained softmax logits top out around 35; the trained kernel scores reach 61,221, and the bias-free ablation, whose $\varepsilon$ cannot learn to grow, reaches 443,508. Nothing in the kernel bounds its value when training is free to grow $\lVert q \rVert$, since $\kappa(q, q) = (\lVert q \rVert^2 + b)^2 / \varepsilon$, and training grew the norms. What survives is the claim that actually matters: none of this is dangerous because there is no exponential downstream. A logit of 35 under softmax overflows float32 without the max trick; a kernel score of $6 \times 10^4$ passes through a ratio of polynomials untouched. The exponential was the hazard. Remove it and scores can be as large as the geometry wants.

The second was the mass channel’s obvious use. If mass measures how much compatibility a query found, low mass should flag tokens the model handles badly, the attention version of the prototype network’s abstention. Measured: the AUROC of row mass predicting next-token correctness is 0.508, 0.509, 0.515 across the three seeds. Null, three times. The channel exists, softmax genuinely cannot represent it, and its naive reading as token confidence carries no signal at this scale. Both facts go in the ledger; whether mass means something else, out-of-distribution text, retrieval quality, head specialization, is an open thread, not a promise.

Scope: all numbers are from scripts/yat_attention.py on Kaggle (kernel quality bundle kgl_blog-yatattn-b-v1 and softmax bundle kgl_blog-yatattn-v1, three seeds each at swept rates; attention-map telemetry bundle kgl_blog-yatattn-b-telem, one seed): character-level tinyshakespeare, parameter-matched GPTs, best-epoch validation loss, kernel and softmax telemetry on held-out batches; the entropy statistics replay the telemetry bundle’s trained maps (scripts/render_yatattn_gifs.py); the learning-rate audit is bundles kgl_blog-yatattn-lrsweep (both variants, four rates, one seed), kgl_blog-yatattn-fair (three seeds at the kernel’s best rate), and kgl_blog-yatattn-b-v1 (three seeds of the canonical per-head-bias kernel), with run-to-run GPU nondeterminism of about 0.01 nats bounding what any single seed can claim. One dataset, one scale, one epsilon; the existence claim is scoped to exactly that.

Where the bridge goes#

The construction closes the loop the two threads opened, and it comes with a road onward. A kernel in the score slot is not just a philosophical upgrade: kernels have an approximation theory, and the SLAY paper (arXiv 2602.04915) builds positive random features for exactly this kernel, the same move Performers made for the softmax surrogate, which turns the quadratic attention matrix into a linear-time product without abandoning the Mercer object. The kernel view of attention stops being a reading and becomes an implementation choice with a dial: exact and quadratic, or featurized and linear. Cheap attention was the survey of that trick from the outside; the Yat version owns it from the inside.

The two threads are one story now. Attention was always a kernel smoother wearing a surrogate; the surrogate can be swapped for the real thing at a measured, small, honestly-reported cost; and everything this series knows about the real thing, its geometry, its prototypes, its theory, its features, is now standing inside the transformer, not beside it.

@misc{bouhsine2026attentionisacompatibilitykernel,
  author       = {Bouhsine, Taha},
  title        = {The Kernel Between the Roles},
  year         = {2026},
  month        = {jul},
  howpublished = {\url{https://tahabouhsine.com/blog/attention-is-a-compatibility-kernel/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}