The Kernel Between the Roles

Two threads of this blog have been walking toward each other for months. One read attention as kernel machinery: a Nadaraya-Watson smoother whose exp(q·k) plays the kernel’s part without being one. The other built a program on a kernel that actually is one, the Yat kernel, and spent it on MLPs, editing, survival models, fixed points. The QK post closed on the meeting point and then refused to build it: the trap, it said, is that a Mercer kernel on tokens is symmetric, which deletes the query and key roles that make attention directional; the right move is to kernelize the compatibility between the roles, $s_{ij} = \kappa(f_Q(x_i), f_K(x_j))$, and that construction deserves a post of its own. This is that post.

What the exponential was hired to do#

Strip attention to its bookkeeping and one requirement stands out. The mechanism turns scores into a weighted average, so the weights must be nonnegative and sum to one. Bilinear scores $q \cdot k$ come out of a matrix multiply signed, positive here, negative there, useless as weights. The exponential is the repair: $e^x$ is positive whatever $x$ was, so wrap the scores and divide by the sum. That is the whole softmax, and it works, and everything distinctive about softmax attention follows from that one repair rather than from anything attention asked for. The scores gain a gauge: shift every logit by a constant and the weights do not move, so the absolute level of compatibility is deleted information, the same shift-invariance that deletes a kernel’s field magnitude in a classifier head. The implementation gains the max-subtraction trick, because $e^{30}$ overflows what floats can hold. And the geometry gains the property the QK post lingered on: $e^{q \cdot k}$ behaves like a kernel smoother operationally while being no Mercer kernel at all.

But suppose the score arrived positive on its own. A Mercer kernel with a nonnegative range needs no repair: weights are $\kappa_{ij} / \sum_j \kappa_{ij}$, done. No exponential, no gauge, no overflow trick, and the smoother is not an analogy anymore, it is the literal Nadaraya-Watson estimator that the first attention post could only read into the formula. The Yat kernel is exactly such an object (arXiv 2605.03262):

Positive semidefiniteness is a fact, not a hope, and the proof fits in a breath: $(q \cdot k + b)^2$ is the quadratic polynomial kernel, $1/(\lVert q - k \rVert^2 + \varepsilon)$ is the inverse multiquadric, and the Schur product theorem says the pointwise product of two kernels is a kernel. The bias deserves a sentence of respect before anything trains, because it is not a convenience constant. Expand the numerator: $(q \cdot k)^2 + 2b\,(q \cdot k) + b^2$, a quadratic, a linear, and a constant term in the alignment. Drop $b$ and only the even term survives, and the paper’s universality theorem fails with it: with $b > 0$ the product kernel’s reproducing-kernel Hilbert space is dense in the continuous functions, a universal kernel, while at $b = 0$ that density is lost. The bias is what buys the theorem. And $\varepsilon$ is the same softening the prototype posts used, capping the kernel’s peak at $(\lVert q \rVert^2 + b)^2 / \varepsilon$ when key meets query. Both scalars are learned, per head, through a softplus, initialized at $\log 2$: each head gets to choose its own floor of compatibility and its own softening, forty-eight parameters in total, and what they learn is itself telemetry. The full apparatus, RKHS, feature map, representer theorem, everything this series built its network program on, rides on top.

And the trap from the QK post is disarmed the way that post prescribed. The kernel is symmetric in its two arguments, but its arguments are not tokens: they are $f_Q(x_i)$ and $f_K(x_j)$, the learned query and key projections. Directionality lives in the roles, $f_Q \neq f_K$, exactly where it lived under softmax. Only the comparison between the roles changes.

What the kernel scores that a dot product cannot#

Before any training, the two compatibility functions disagree about geometry, and the disagreement is visible. A bilinear score rewards one thing: projection onto the query’s direction. It grows without bound along that ray and cannot tell a key sitting on the query from a key ten times farther out along the same line. The Yat kernel multiplies alignment by proximity: its numerator wants the directions to agree, its denominator wants the vectors near each other, and its peak sits at the query itself.

This is the prototype geometry this series built for neurons transplanted into attention: under a Yat score, a key is not a direction to project onto but a place, and a query attends hardest to keys that are near it in the projected space. Whether a language model wants that inductive bias is precisely what an existence-proof experiment is for.

What each normalizer forgets#

The second structural difference needs no training either. Softmax’s shift gauge means the row of weights is all that survives; the level of the scores, how compatible the best match actually was, is deleted by construction. The kernel’s normalization also produces weights summing to one, but the raw row sum, $m_i = \sum_j \kappa_{ij}$, is a real nonnegative number that survives as a separate channel: how much total compatibility this query found, before normalization spent it.

The mass channel is the attention version of the kernel field magnitude that a bounded network keeps beneath its softmax, the quantity the prototype network’s abstention story was built on. Structurally, it is free information. What it is worth in a transformer, the run below measures, and the measurement deserves to be reported exactly as it came out.

The run#

Two character-level GPTs on Shakespeare, identical in every respect except the score slot: 6 layers, 4 heads, model width 192, context 128. The softmax model computes $\mathrm{softmax}(q \cdot k / \sqrt{d})$ at 2,719,169 parameters; the Yat model computes $\kappa / \sum \kappa$ with its per-head learned $b$ and $\varepsilon$, 2,719,217 parameters, the difference being exactly those forty-eight scalars, and contains no softmax anywhere in its attention. Each side trains at its own swept learning rate, 3e-4 for softmax and 3e-3 for the kernel, per the fairness audit below. Three seeds each, same data order, 12,000 steps, best validation loss (scripts/yat_attention.py, bundles kgl_blog-yatattn-v1 and kgl_blog-yatattn-b-v1).

Softmax finishes at 1.4923 ± 0.0060 nats per character, the kernel at 1.5089 ± 0.0020, a 1.1 percent gap. The learning rates deserve their provenance, because this series has asked the fairness question before, in the survival-model trial: each side must train at its own best rate. A full sweep, both mechanisms at 3e-4, 1e-3, 3e-3, and 1e-2, found softmax’s optimum at 3e-4 and the kernel’s at 3e-3, the roughly tenfold ratio the Yat training experience predicts. The sweep’s shape is worth more than its argmaxes: softmax’s quality lives on a cliff, best in one narrow window, degraded at 1e-3, and at 1e-2 it diverges outright to 2.58 nats, an untrained model, while the kernel walks the entire 33-fold range without leaving a one percent band. An exponential amplifies whatever scale the gradients hand it, so softmax’s learning rate carries the job of keeping the logits in the survivable window; a ratio of polynomials normalizes whatever arrives. The kernel spends no hyperparameter budget on staying alive.

And the bias earns its theorem-fee in the ablation. Set $b = 0$, the non-universal special case, and three-seed quality slips to 1.5131 ± 0.0071, a loss at the boundary of run-to-run noise, but the score telemetry moves decisively: the ablation’s maximum trained score runs to 443,508 against the full kernel’s 61,221. There is also a routing option the ablation deletes outright: at $b = 0$ an orthogonal query-key pair scores exactly zero regardless of how near each other the two vectors sit. Universality is not an abstraction fee, it is these degrees of freedom.

And the forty-eight scalars use their freedom. Exported from the trained model, the biases have wandered from their $\log 2$ initialization across the range 0.22 to 3.40, every layer holding heads that raised their compatibility floor and heads that lowered it. The softenings split more dramatically: most heads drive $\varepsilon$ toward zero, sharpening their kernel’s peak toward a nearest-neighbor router, while two late-layer heads grow $\varepsilon$ to 6.4 and 12.7, spreading themselves toward global averagers. Per-head kernel scalars are not a numerical nicety; they are each head choosing what kind of statistician to be, and the choice is readable off two vectors of numbers in a way no $W_Q W_K^\top$ ever is.

What the sweep found instead is a difference worth more than the gap. Softmax’s quality lives on a cliff: best at 3e-4, already degraded at 1e-3, and at 1e-2 it diverges outright, 2.58 nats, an untrained model. The kernel walks the entire 33-fold range and never leaves the 1.50 to 1.52 band. At every learning rate except softmax’s own optimum, the two mechanisms tie or the kernel wins. The mechanism is the one this post already named: an exponential amplifies whatever scale the gradients hand it, so the logits must be kept in a narrow band and the learning rate does that job; a ratio of polynomials normalizes whatever scale arrives. The kernel spends no hyperparameter budget on staying alive. Per the standing frame of this whole program: the number is not a victory and is not trying to be one. The claim is the existence proof, the same one the solve post made for kernel machines and the Yat MLP posts made for networks: a transformer whose routing is a genuine Mercer-kernel smoother, with no exponential in the path, trains by plain gradient descent to within a hair of the mechanism that has monopolized the architecture since 2017, while keeping the kernel’s structure, the geometry, the mass channel, the theory, intact underneath.

And here is what the trained routing looks like, side by side on one Shakespeare window, from the telemetry run’s checkpoints:

Similar, but not the same, and the difference is measurable rather than aesthetic. Score each trained query row by its normalized entropy, zero for a row that stakes everything on one key, one for a row spread evenly, and the two mechanisms separate cleanly: the softmax model averages 0.48, with 22 percent of its rows sharpened nearly to a single key; the kernel model averages 0.73, with 0.2 percent of rows that sharp and 37 percent diffuse. This is not a training accident, it is the arithmetic of the two normalizers. An exponential turns score gaps into weight ratios that grow without limit, which is why softmax is rightly described as a soft argmax: give one key a few logits of advantage and it takes essentially everything. A ratio of polynomials cannot do that; for one key to take 99 percent of the row, its kernel value must exceed the sum of all others hundredfold on the raw scale, not the log scale. The kernel buys its geometry at the price of concentration, and that price is a plausible reading of the 1.4 percent: character-level language leans hard on copy-like heads that want to be nearly one-hot. It also names the obvious dial for closing the gap, raising the numerator’s power, $(q \cdot k)^{2m}$, which sharpens the kernel’s peak while every structural property above survives.

The exponential’s second job#

That last sentence pointed at a dial, and the dial has a name this post has been avoiding: the exponential itself. Positivity was the job we opened by retiring, and the kernel does retire it. But an exponential does a second thing a bare ratio cannot, and the entropy gap is its fingerprint. It sharpens: it turns score gaps into weight ratios that grow without bound. Nothing stops the kernel from keeping its geometry and borrowing that sharpener back, normalizing its already-positive scores with a softmax instead of an $\ell_1$ sum. So we ran it, same kernel, same parameters, $\mathrm{softmax}(\kappa)$ in place of $\kappa / \sum \kappa$:

On the full kernel the sharpener closes the gap outright: 1.4968 sits inside run-to-run noise of standard softmax’s 1.4923. Character-level language wants near-one-hot copy heads, the kernel’s $\ell_1$ normalization cannot make them, and the exponential can, now on top of a Mercer score rather than a bilinear one. On the projection-free tier without V it helps again. On the middle tier it does neither: 1.5559, with a seed spread ten times the others’. The sharpener is a real tool with a real failure mode, not a free upgrade, but where it helps it costs zero parameters and no new geometry, only a different normalizer over the same kernel.

Which raises the question you would ask next: if the exponential is a detachable sharpener, can a model trained one way be run the other, borrowing sharpness or the mass channel at inference for free? The scores are the same object under both normalizers, so the swap is well defined. It is also a disaster, symmetrically. An $\ell_1$-trained model forced to softmax at test collapses from 1.75 to 3.18 nats; a softmax-trained model forced to $\ell_1$ collapses from 1.50 to 3.23. Both land at the loss of an untrained network, in both directions.

The reason is written in the scalars each model learned:

An $\ell_1$-trained head sharpens from inside the kernel, collapsing its $\varepsilon$; an exp-trained head sharpens outside, at the normalizer, and leaves the kernel soft. The scores each produces are calibrated for the normalizer it trained under and nowhere else. You get the mass channel and the learning-rate robustness by training under $\ell_1$; you get the last percent of quality by training under softmax; you cannot borrow one for the other after the fact. The exponential’s two jobs separate cleanly on paper and stay married in a trained network.

Two of our beliefs, measured#

An existence proof earns its keep by what it refuses to fudge, and this run falsified two things we expected to write.

The first was the word bounded. The Yat kernel is bounded for bounded inputs, and we expected the kernel scores to stay tame while softmax logits roamed. The opposite happened. The trained softmax logits top out around 35; the trained kernel scores reach 61,221, and the bias-free ablation, whose $\varepsilon$ cannot learn to grow, reaches 443,508. Nothing in the kernel bounds its value when training is free to grow $\lVert q \rVert$, since $\kappa(q, q) = (\lVert q \rVert^2 + b)^2 / \varepsilon$, and training grew the norms. What survives is the claim that actually matters: none of this is dangerous because there is no exponential downstream. A logit of 35 under softmax overflows float32 without the max trick; a kernel score of $6 \times 10^4$ passes through a ratio of polynomials untouched. The exponential was the hazard. Remove it and scores can be as large as the geometry wants.

The second was the mass channel’s obvious use. If mass measures how much compatibility a query found, low mass should flag tokens the model handles badly, the attention version of the prototype network’s abstention. Measured: the AUROC of row mass predicting next-token correctness is 0.508, 0.509, 0.515 across the three seeds. Null, three times. The channel exists, softmax genuinely cannot represent it, and its naive reading as token confidence carries no signal at this scale. Both facts go in the ledger; whether mass means something else, out-of-distribution text, retrieval quality, head specialization, is an open thread, not a promise.

Scope: all numbers are from scripts/yat_attention.py on Kaggle (kernel quality bundle kgl_blog-yatattn-b-v1 and softmax bundle kgl_blog-yatattn-v1, three seeds each at swept rates; attention-map telemetry bundle kgl_blog-yatattn-b-telem, one seed): character-level tinyshakespeare, parameter-matched GPTs, best-epoch validation loss, kernel and softmax telemetry on held-out batches; the entropy statistics replay the telemetry bundle’s trained maps (scripts/render_yatattn_gifs.py); the learning-rate audit is bundles kgl_blog-yatattn-lrsweep, kgl_blog-yatattn-fair, and kgl_blog-yatattn-b-v1; the projection-free tiers are bundles kgl_blog-goat-lrsweep (+-low, 2, -hi: the bracketing sweep, 3e-4 to 1e-1), kgl_blog-goat-v-v1, kgl_blog-goat-nov-v1 (three seeds at each bracketed optimum), and kgl_blog-goat-telem; the exponential-normalizer arms and the crossover swaps are bundles kgl_blog-yatexp-* and kgl_blog-swap-* (the swap collapses are single-seed final-step evaluations, where the ~1.7-nat drop is unambiguous). Run-to-run GPU nondeterminism of about 0.01 nats bounding what any single seed can claim. One dataset, one scale, one epsilon; the existence claim is scoped to exactly that.

Do we even need Q and K?#

Follow the argument one step further and it walks somewhere the QK post did not anticipate. That post defended the projections by pointing at the score: a bilinear form $x_i^\top W_Q W_K^\top x_j$ is its matrix, delete the projections and only the raw dot product remains, which can express almost nothing. But that defense rests entirely on the score being bilinear. The Yat kernel is not. It compares vectors through alignment, distance, a learned floor, and a learned softening, a genuinely nonlinear compatibility that exists before any projection is learned. So the projections’ job description, giving a linear form something to say, is gone. Whether anything else needed them is not a matter of argument anymore; it is a run.

Strip them and one requirement appears immediately, with a one-line derivation. Let the kernel compare raw residual-stream head slices, so query and key are the same vector family. At $u' = u$ the distance in the denominator hits its floor $\varepsilon$ while the numerator sits at its peak, so $\kappa(u, u) = (\lVert u \rVert^2 + b)^2 / \varepsilon$ is the largest entry of its row. Normalize and every token attends mostly to itself: the layer collapses toward a pointwise re-scaling that mixes nothing. The repair costs nothing in a language model: make the causal mask strict, $j < i$, so a token may not attend to itself and its own information travels on the residual stream instead. And notice what that mask also settles: the QK post’s worry that a symmetric kernel deletes directionality. The mask is not symmetric. Token thirty may read token seven; token seven cannot read token thirty. The roles the projections used to carry live in the geometry of time.

Two tiers of removal, three seeds each, each at its own bracketed learning rate:

Read the last row slowly, because it is the strangest object in this post: an attention layer whose only parameters are the output projection and two scalars per head, whose scores are a fixed kernel on raw hidden states, which caches nothing at inference, and which lands 2.8 percent behind tuned softmax with a quarter of the attention stack’s parameters deleted. The middle rows say the rest: removing Q and K costs 1.4 percent against the full kernel, and then removing V costs 0.3 percent more, inside overlapping seed noise, so once the projections that feed the score are gone, the value projection is barely earning its keep either; the output projection absorbs its role.

The learning-rate landscape of these tiers deserves its own sentence, because it extends the sweep’s earlier finding into a law-like progression. Each mechanism’s bracketed optimum: softmax 3e-4, the full kernel 3e-3, the projection-free tiers 1e-2 and 3e-2. Every piece of learned linear algebra removed moves the optimum up an octave or two, until the leanest model trains best at one hundred times the rate where softmax’s optimum sits, and three times past the rate where softmax diverges outright. The sweep also kept us honest twice over: the projection-free tiers’ low-rate cells, measured rather than inferred, came back better than their mid-rate neighbors, a non-monotone pocket the extrapolate-the-trend shortcut would have papered over.

The forty-eight scalars tell their own story about what removal changed. On projected inputs the learned softenings mostly shrank toward zero, heads sharpening themselves; on raw slices they do the opposite, growing to a mean of 13.8 with individual heads as high as 50, while the biases spread across everything from zero to 10.2. Deprived of projections that could rescale the stream, the heads move the same knobs the other way and buy their stability inside the kernel itself. The routing stays diffuse (entropy 0.72, matching the full kernel’s 0.73), so the character of kernel attention survives the amputation; only its internal accounting rearranges.

The gap has a direction worth naming without overclaiming. Between the full kernel and the projection-free tiers sits everything a learned Q and K can still do that raw slices cannot: choose which subspace of the residual stream the compatibility should care about. One or two percent of quality is apparently what that choice is worth at this scale. What the projections are demonstrably not needed for is the thing attention is named after: with a kernel in the score slot, routing, structure, and trainability all survive their removal.

Where the bridge goes#

The construction closes the loop the two threads opened, and it comes with a road onward. A kernel in the score slot is not just a philosophical upgrade: kernels have an approximation theory, and the SLAY paper (arXiv 2602.04915) builds positive random features for exactly this kernel, the same move Performers made for the softmax surrogate, which turns the quadratic attention matrix into a linear-time product without abandoning the Mercer object. The kernel view of attention stops being a reading and becomes an implementation choice with a dial: exact and quadratic, or featurized and linear. Cheap attention was the survey of that trick from the outside; the Yat version owns it from the inside.

The two threads are one story now. Attention was always a kernel smoother wearing a surrogate; the surrogate can be swapped for the real thing at a measured, small, honestly-reported cost; and everything this series knows about the real thing, its geometry, its prototypes, its theory, its features, is now standing inside the transformer, not beside it.

@misc{bouhsine2026attentionisacompatibilitykernel,
  author       = {Bouhsine, Taha},
  title        = {The Kernel Between the Roles},
  year         = {2026},
  month        = {jul},
  howpublished = {\url{https://tahabouhsine.com/blog/attention-is-a-compatibility-kernel/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}