Machine learning research notes by Taha Bouhsine — neural network interpretability, kernel methods, contrastive learning, attention mechanisms, RKHS, and transformer architectures. Long-form pieces with interactive visualisations.https://tahabouhsine.com/https://tahabouhsine.com/survival-model-on-trial-jax-flax-nnx/https://tahabouhsine.com/survival-model-on-trial-jax-flax-nnx/A runnable companion to the survival-model trial: the Yat DeepSurv trunk in Flax NNX, the Cox partial-likelihood loss, an LR-fair per-model training loop with best-epoch selection, the concordance / integrated-Brier / time-dependent-AUC evaluation, and the classical baselines wired through sksurv and lifelines. Every figure is a real number from a real run across five datasets, with the prototypes and the risk stratification captured as they form over training.Thu, 09 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/transformers-with-a-velocity-ledger-jax-flax-nnx/https://tahabouhsine.com/transformers-with-a-velocity-ledger-jax-flax-nnx/A runnable companion: the pre-norm Transformer block as a forward-Euler step, then the residual-stream velocity ledger as one line of Flax NNX state (mu = 0 recovers plain), the ngpt-lite retraction variant, best-val early-stopped training, and the depth telemetry (path length and turning angle per sub-update). Four parameter-matched char-level GPTs that tie on quality and split on dynamics: the ledger's residual-stream path is a third as long and half as sharp.Thu, 09 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/survival-model-on-trial/https://tahabouhsine.com/survival-model-on-trial/A classical kernel machine is beautiful and needs a solve that will not minibatch or compose. A deep net trains on anything and its risk score is a fog. What if one thing had the training of a net and the theory of a kernel? We put a deep Yat-kernel survival model on trial across five real datasets against Cox, penalized Cox, and Random Survival Forest: it trains with plain gradient descent, lands in the pack on the concordance index, and, because its units are genuine Mercer kernels, inherits exact attribution, calibration, editing, and bounded out-of-distribution response the others cannot give. Where it costs a point or two, we show exactly where.Thu, 09 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/transformers-with-a-velocity-ledger/https://tahabouhsine.com/transformers-with-a-velocity-ledger/A pre-norm Transformer's residual stream is forward Euler: x += Attn(norm x); x += MLP(norm x). So D1's whole dictionary transfers, and the same question follows: does a velocity ledger in the residual stream buy in a Transformer what it bought in a ResNet? The answer splits. On quality, four variants tie. On dynamics, the ledger changes everything: the residual-stream path through depth gets dramatically shorter and straighter, reaching the same answer by a calmer journey. Same destination, gentler road.Thu, 09 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/a-risk-model-that-names-its-reasons-jax-flax-nnx/https://tahabouhsine.com/a-risk-model-that-names-its-reasons-jax-flax-nnx/A runnable companion: the Cox partial-likelihood loss, a standard DeepSurv, and a Yat-kernel DeepSurv whose log-risk decomposes into prototype patients, all in Flax NNX. Concordance evaluation, exact convex attribution, cohort deletion, and OOD abstention as short array operations. Every number is from a real run on METABRIC breast-cancer survival.Sun, 05 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/a-risk-model-that-names-its-reasons/https://tahabouhsine.com/a-risk-model-that-names-its-reasons/A survival model tells an oncologist a patient is high-risk, and she has to act on the number without being able to ask why. What would it take for the risk score to name its reasons? We build a Yat-kernel DeepSurv on breast-cancer survival, match a standard DeepSurv on concordance, and get a risk score that decomposes exactly into the prototype patients this one resembles, a model you can read, audit, and edit.Sun, 05 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/calibration-of-a-bounded-net-jax-flax-nnx/https://tahabouhsine.com/calibration-of-a-bounded-net-jax-flax-nnx/A runnable companion: build the matched Yat and ReLU MLPs in Flax NNX with the same softmax head, then measure their honesty. The reliability diagram and ECE, temperature scaling fit on a held-out split, NLL and Brier, and the two out-of-distribution channels, kernel-field magnitude versus softmax confidence, all in JAX with every number from a real three-seed run.Sat, 04 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/depth-by-construction-jax-flax-nnx/https://tahabouhsine.com/depth-by-construction-jax-flax-nnx/A runnable companion: build a whole second feature layer by hand in JAX, on top of the hand-built first. Named min-AND combinations of layer-1 edges (junctions, continuations, bends, stripes) feed the same constructed Yat head, no training anywhere. It reproduces the flat rung: 83.3% at layer 1, 82.9% with both, 78.8% from relations alone, and counts the combinatorial wall of 224 pairwise and 4,630 three-way types where construction stops.Sat, 04 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/distillation-is-kernel-transfer-jax-flax-nnx/https://tahabouhsine.com/distillation-is-kernel-transfer-jax-flax-nnx/A runnable companion: the five-run distillation experiment in JAX/Flax NNX. Train a teacher CNN, extract its class-similarity kernel S = E[softmax(z/T) softmax(z/T)ᵀ], train a student on nothing but pairwise relations (no labels, no soft targets), and measure it against the label ceiling and the random floor with a linear and a nearest-centroid probe. Every number is from a real run, with six GIFs that animate the kernel assembling, the temperature dial, the handoff, the spectrum inheritance, the probe race, and the inherited mistakes.Sat, 04 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/edit-a-fixed-point-jax-flax-nnx/https://tahabouhsine.com/edit-a-fixed-point-jax-flax-nnx/A runnable companion: build the weight-tied Yat equilibrium operator in Flax NNX, then teach a class by appending rows to the readout (F untouched, exact) or into the shared dynamics (one paste, present at every depth), measure the contraction certificate with power iteration and bisect one gain to restore it, audit the drift of 520 old fixed points, watch a layer-only edit evaporate, and forget by masking. Every number is from a real run.Sat, 04 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/momentum-resnet-jax-flax-nnx/https://tahabouhsine.com/momentum-resnet-jax-flax-nnx/A runnable companion: the residual block as a forward-Euler step, then the momentum residual network as a Flax NNX module with one extra state, a velocity the blocks write into. Train both on the rings task a first-order flow cannot separate exactly, watch the training crystallize, and run the trained network exactly backward until floating point, amplified by 1/mu per layer, steals the past.Sat, 04 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/calibration-of-a-bounded-net/https://tahabouhsine.com/calibration-of-a-bounded-net/A network hands you a probability with every answer, and the number is the part you act on. So when this series' bounded, self-explaining kernel network says 80%, is that a measurement or a mood? Five posts of evidence say it should be the honest one. This post puts that reputation through a lie-detector test, reliability diagrams, expected calibration error and temperature scaling against a matched ReLU MLP on Fashion-MNIST, and what the test found is the post.Sat, 04 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/depth-by-construction/https://tahabouhsine.com/depth-by-construction/One hand-built feature layer matched a trained backbone at 83.3% on Fashion-MNIST, and real networks are deep. Conveniently, the recipe for a second layer has been on the shelf for half a century: vision science says edges assemble into junctions, continuations, bends and stripes. This post takes the recipe down and follows it, builds layer 2 entirely by hand with every dimension still nameable in one sentence, and measures exactly where construction stops, and why.Sat, 04 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/distillation-is-kernel-transfer/https://tahabouhsine.com/distillation-is-kernel-transfer/Knowledge distillation has a standing puzzle: Hinton's student recognized 98.6% of the digit 3s in the test set after training on a transfer set with every 3 deleted. An answer key cannot do that, so what actually crosses the wire? This post gives dark knowledge a data type, a class-similarity kernel, and runs the experiment that isolates it: a student trained on nothing but pairwise relations, no labels, no soft targets, no class names, measured against the label-trained ceiling and the random floor. With live experiments: watch the kernel accumulate from single outputs, turn the temperature knob on how much geometry leaks, train a relational student in the page, and watch whose spectrum the student grows into.Sat, 04 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/edit-a-fixed-point/https://tahabouhsine.com/edit-a-fixed-point/One post taught and forgot classes by editing rows of a Yat network, with proofs that nothing else moved. Another melted the stack of layers into a single operator iterated to a fixed point. This is the collision. Every one of those editing proofs rested on a pasted row entering the score once, as one term in one sum, and in an equilibrium network there is no once: whatever you paste is applied at every depth and fed back into its own input, and every fixed point is free to drift. So did melting the stack melt the editability? This post pastes, deletes, and measures: every guarantee that survives is either proved inside the recursion or measured against the real run, fixed point by fixed point.Sat, 04 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/skip-connections-are-half-of-newton/https://tahabouhsine.com/skip-connections-are-half-of-newton/A residual block x + F(x) is one forward-Euler step: depth is time, the block is a velocity, position moves directly. That is half of Newtonian mechanics. A planet does not update position from force; force updates velocity, velocity updates position, and that split is why orbits are stable. So what does the missing half cost a deep network? We let the physics make three predictions about trained networks, then check all three live in the page. One of them comes back stranger than we wrote it.Sat, 04 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/regularization-is-a-price-list-jax-flax-nnx/https://tahabouhsine.com/regularization-is-a-price-list-jax-flax-nnx/A runnable companion to the price-list post: kernel ridge in JAX, the representer solve (K + lambda I) alpha = y, the RKHS-norm bill alpha^T K alpha, the effective dimension d_eff = sum lambda_k/(lambda_k + lambda) from the Gram spectrum, and a generalization sweep that draws the U-curve. Every number and every figure is from one analytic solve, no gradient descent.Thu, 02 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/regularization-is-a-price-list/https://tahabouhsine.com/regularization-is-a-price-list/The representer theorem says the optimal weight is a sum over prototypes, but it does not explain why that sum generalizes. The answer is the RKHS norm: a price list that charges each prototype by its eigenvalue, and regularization is just tightening the budget. Four panels show the knob turning.Thu, 02 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/your-network-is-a-fixed-point-jax-flax-nnx/https://tahabouhsine.com/your-network-is-a-fixed-point-jax-flax-nnx/A runnable companion: build the Yat deep-equilibrium network in JAX/Flax NNX. One shared operator F(z;x)=tanh(A·φ_W(z)+Ux+z0), solved to its fixed point by damped iteration, trained not by backprop-through-iterations but by implicit differentiation, the adjoint (I−Jᵀ)u=∂L/∂z* run by the same contraction. Plus the weight-tied maze operator that extrapolates from 11×11 to 27×27 by iterating longer. Every number is from a real run: 98.2% on two moons from 1700 shared parameters, ‖J‖ 0.66/0.92, 99.5% on mazes far larger than training.Wed, 01 Jul 2026 17:00:00 GMThttps://tahabouhsine.com/your-network-is-a-fixed-point/https://tahabouhsine.com/your-network-is-a-fixed-point/A deep network makes you choose its depth before you have seen the problem, and gives every layer its own weights. Share one Yat-kernel operator across all of them instead, and the stack collapses into a single equation: the answer is the fixed point the state settles into. Training makes that operator a contraction, so the settling point is unique and reached from anywhere, the network decides its own depth per input, and the same twenty-four prototypes describe the computation at every step. 98.2% on two moons from 1700 parameters shared across all depth.Wed, 01 Jul 2026 00:00:00 GMThttps://tahabouhsine.com/mlp-block-is-a-representer-theorem-jax-flax-nnx/https://tahabouhsine.com/mlp-block-is-a-representer-theorem-jax-flax-nnx/A runnable companion: build a transformer whose feed-forward block is a Yat kernel, so the FFN is exactly a representer sum over learned key-value memory slots. Train it on tinyshakespeare, then do four things you cannot do to an opaque ReLU FFN: read each memory slot, attribute an output to the slots that wrote it, edit one slot and watch generation change, and read off when the memory is out of its depth.Sat, 27 Jun 2026 17:00:00 GMThttps://tahabouhsine.com/mlp-block-is-a-representer-theorem/https://tahabouhsine.com/mlp-block-is-a-representer-theorem/After the 3Blue1Brown attention video you can read half a transformer: you can see which token attends to which. The other half, the MLP block, stays a black box. But attention is legible because it is a kernel, a vote by similarity, and if you make the MLP a kernel too, its output becomes the same thing: a representer-theorem vote over learned prototypes. Then the whole transformer explains itself.Sat, 27 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/what-can-a-weight-be-jax-flax-nnx/https://tahabouhsine.com/what-can-a-weight-be-jax-flax-nnx/A runnable companion: compute the price list of a kernel in JAX. The eigenvalues are the kernel's spectral density, found with an FFT; the RKHS norm of a weight is a sum over them. A corner is affordable only under a Sobolev kernel, and the same numbers place the Yat kernel: universal and smooth, roomier than a Gaussian but not a Sobolev space.Fri, 26 Jun 2026 17:00:00 GMThttps://tahabouhsine.com/what-can-a-weight-be/https://tahabouhsine.com/what-can-a-weight-be/Once a kernel gives a weight a home, a second question follows: what is the weight allowed to be? Not all reproducing kernel Hilbert spaces are the same. A Sobolev space lets the weight have a sharp corner; a Gaussian's space forbids it; on normalized data the home is a sphere graded by spherical harmonics. A kernel is secretly a price list for roughness, and that list decides everything. Four interactive panels.Fri, 26 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/where-does-a-weight-live-jax-flax-nnx/https://tahabouhsine.com/where-does-a-weight-live-jax-flax-nnx/A runnable companion: build the representer-theorem weight in JAX. A positive-definite kernel, the Gram matrix, a single linear solve for the coefficients, and the weight comes out as a combination of the data, f = sum alpha_i k(x_i, .). A linear weight cannot separate nested rings; the placed kernel weight does, read purely through the kernel as a similarity-weighted vote of the data.Thu, 25 Jun 2026 17:00:00 GMThttps://tahabouhsine.com/where-does-a-weight-live/https://tahabouhsine.com/where-does-a-weight-live/A standard neuron's weight and its input never actually meet: one is a point you can see, the other an arrow off in its own space, joined only by a shadow. This is what a reproducing kernel Hilbert space fixes: it gives input and weight one shared address, where the optimal weight is built from the data itself and sits right next to it. Four interactive panels.Thu, 25 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/handbuilt-features-jax-flax-nnx/https://tahabouhsine.com/handbuilt-features-jax-flax-nnx/A runnable companion: build the training-free image classifier from the post in JAX. The feature extractor is pure JAX (Sobel gradients, orientation binning, patch pooling); the classifier is a Flax NNX module holding k-means prototypes that votes with the Yat kernel. Nothing is trained, and it reproduces the 83.3% on Fashion-MNIST.Wed, 24 Jun 2026 17:00:00 GMThttps://tahabouhsine.com/you-dont-have-to-train-the-features/https://tahabouhsine.com/you-dont-have-to-train-the-features/The last post trained a backbone and built the classifier by hand. This one builds the features by hand too: oriented-edge and corner detectors pooled over a grid of patches, the way computer vision worked for decades. Feed those to the same constructed Yat head and, with nothing trained anywhere, it matches the trained backbone on Fashion-MNIST point for point, within a couple of points of a fully trained network. The whole network is hand-built and readable end to end.Wed, 24 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/train-the-features-jax-flax-nnx/https://tahabouhsine.com/train-the-features-jax-flax-nnx/A runnable companion: train a small conv backbone in Flax NNX, then on its frozen features build a constructed Yat head with no gradient steps and compare. The constructed head lands within a couple of points of the trained one, and even a random backbone's features sort at 73% while its trained head is at chance.Tue, 23 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/train-the-features/https://tahabouhsine.com/train-the-features/Leave a convolutional network's weights at their random starting values and build a Yat head on its features by hand: the trained head on that random backbone sorts at chance while the constructed one reaches 74%. On a properly trained backbone the constructed head reaches 83.2% against 85.7% for the trained one. The accuracy lives in the representation; the classifier, and its edits, are furniture you place. This maps the boundary between what you must optimize and what you can construct.Mon, 22 Jun 2026 23:00:00 GMThttps://tahabouhsine.com/edit-a-network-jax-flax-nnx/https://tahabouhsine.com/edit-a-network-jax-flax-nnx/A runnable companion: build the prototype Yat-MLP in Flax NNX, then add a class by concatenating a few prototype rows and forget a class by masking them out, with no gradient steps. Class-incremental learning that matches a from-scratch build, and exact machine unlearning, both as array edits you can read. Every number is from a real run on Fashion-MNIST.Mon, 22 Jun 2026 18:00:00 GMThttps://tahabouhsine.com/edit-a-network-by-hand/https://tahabouhsine.com/edit-a-network-by-hand/If a neuron is a labelled picture, a classifier is a list of them, and a list is something you edit. Add a class to a trained-free Yat-kernel network by placing twenty pictures, and it recognizes that class at 95% with zero gradient steps. Delete a class by removing its pictures, and it is forgotten exactly, the other classes untouched. Class-incremental learning with no penalty and machine unlearning that is instant and exact, both falling out of the architecture rather than bolted on.Mon, 22 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/yat-mlp-fmnist-jax-flax-nnx/https://tahabouhsine.com/yat-mlp-fmnist-jax-flax-nnx/A runnable companion: build the prototype MLP from the post in Flax NNX, train it on Fashion-MNIST, and watch the neurons. Pull the prototypes out as images, read a prediction as a vote over pictures, see the model abstain on out-of-distribution digits, check that random-init prototypes classify but stay noise, and track the prototypes migrating through a UMAP fit on the dataset as they train.Sun, 21 Jun 2026 17:00:00 GMThttps://tahabouhsine.com/your-neuron-is-a-picture/https://tahabouhsine.com/your-neuron-is-a-picture/Why should a neuron store a direction when it could store a thing? A direction is not a referent you can point at, which is why MLPs are opaque. Put the Yat kernel where the activation was, train on Fashion-MNIST, and every neuron becomes a prototype that lives in pixel space, literally a picture, so the network reads its own predictions: this looks like that, no saliency method required.Sun, 21 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/yat-mlp-jax-flax-nnx/https://tahabouhsine.com/yat-mlp-jax-flax-nnx/A runnable companion to What a Finite Kernel Buys an MLP: build a layer whose unit is the Yat kernel instead of a linear map plus an activation, assert it is positive definite and nonnegative, write down its exact finite feature map, train it end-to-end on two moons with no activation function, and measure the lazy-loading sparsity, the bounded off-distribution response, the RKHS capacity, and the force field that pulls each prototype onto its data.Thu, 18 Jun 2026 17:00:00 GMThttps://tahabouhsine.com/what-a-finite-kernel-buys-an-mlp/https://tahabouhsine.com/what-a-finite-kernel-buys-an-mlp/Replace the activation function with a finite, explicit, positive-definite kernel, the Yat kernel, and an MLP stops being a stack of linear maps glued by a nonlinearity. It becomes a kernel machine, with locality, attribution, geometry, capacity control, and a feature map you can write down.Thu, 18 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/three-states-of-information-jax/https://tahabouhsine.com/three-states-of-information-jax/A runnable companion to The Three States of Information: train tiny models in JAX and measure the three states directly: the feature-covariance spectrum collapsing from high-rank (random) to a C−1-mode frame (structured), the distributional simplicity bias that fits low-order structure first (organized), the neural-collapse simplex where class-mean cosines lock onto −1/(C−1), and the alignment/uniformity split of contrastive learning running on two separate clocks. Four live JAX visualizations, every number an eigenvalue or a loss.Sun, 07 Jun 2026 19:45:00 GMThttps://tahabouhsine.com/three-states-of-information/https://tahabouhsine.com/three-states-of-information/Representations learned by a network pass through three states, like matter: random (high-entropy, no structure), organized (clusters, local order), and structured (a maximally-separated simplex, global order). The transitions between them are exactly the loss plateaus you see when training: the flat stretch is where the representation reorganizes before that reorganization shows up in the loss. Built from live in-browser training runs.Sun, 07 Jun 2026 19:30:00 GMThttps://tahabouhsine.com/latent-on-the-spectrum-jax/https://tahabouhsine.com/latent-on-the-spectrum-jax/A runnable companion to Latent on the Spectrum: build a codebook as the spectral embedding of a label kernel in JAX (classical MDS with square-root eigenvalue scaling), watch a flat spectrum become the simplex and a graded one become the horseshoe, measure kernel-target alignment, split a representation into its between-class prototype frame and within-class information spectrum, and watch neural collapse grind the information to zero.Sun, 07 Jun 2026 19:15:00 GMThttps://tahabouhsine.com/latent-on-the-spectrum/https://tahabouhsine.com/latent-on-the-spectrum/The regular simplex is the perfect codebook only when classes are strangers, and real labels are not strangers. A latent space is a lossy, finite-dimensional encoding of a label-similarity kernel: the codebook is the top eigenmodes of that kernel, the information rides in the modes below them, and the Welch bound sets the geometry of that channel. A follow-up to the Welch-bound post with live in-browser experiments: steer a codebook from simplex to taxonomy, spend a dimension budget, watch neural collapse grind the information spectrum to zero, read dark knowledge off a wandering feature, and see a structured codebook make better mistakes.Sun, 07 Jun 2026 19:00:00 GMThttps://tahabouhsine.com/qk-projections-jax-flax-nnx/https://tahabouhsine.com/qk-projections-jax-flax-nnx/A runnable companion to Why Attention Needs Q and K Projections: build scaled dot-product attention with separate query and key projections in Flax NNX, pull the bilinear form B = W_Q W_Kᵀ out of the module, split it into a symmetric metric and an antisymmetric directed part, wire a toy induction head, add RoPE, and measure the low-rank budget and the gauge freedom, all in plain JAX.Thu, 04 Jun 2026 19:45:00 GMThttps://tahabouhsine.com/why-attention-needs-qk-projections/https://tahabouhsine.com/why-attention-needs-qk-projections/The dot product in attention is not enough by itself. Without learned query and key projections, attention can only compare tokens in the residual stream’s native geometry. With a shared projection it learns a symmetric metric. With separate Q and K projections, the score becomes a learned bilinear form x_iᵀW_QW_Kᵀx_j: directional, role-aware, low-rank, and different per head. That bilinearity is what lets attention ask one kind of question and let tokens advertise another kind of answer.Thu, 04 Jun 2026 19:30:00 GMThttps://tahabouhsine.com/convex-readout-jax-flax-nnx/https://tahabouhsine.com/convex-readout-jax-flax-nnx/A runnable companion to The Readout is a Convex Combination of Prototypes: read the columns of W_out as output prototypes in Flax NNX, measure the convex/conic/affine/linear regimes numerically, then build a Nadaraya–Watson kernel readout that is convex by construction (nonnegative weights that sum to one, a point that never leaves the prototype hull), with the nonnegativity-vs-positive-definiteness distinction checked in code.Thu, 04 Jun 2026 19:15:00 GMThttps://tahabouhsine.com/readout-as-convex-combination/https://tahabouhsine.com/readout-as-convex-combination/The second linear map in a transformer MLP is not just a projection. If the hidden activations are nonnegative and normalized, W_out reads the active neurons as a convex combination of output prototypes. Two independent constraints, nonnegativity and summing to one, sort the readout into four regimes: convex, conic, affine, and linear. This reframes the MLP readout as the same object that makes attention legible (a weighted sum over named basis elements), connects it to feed-forward key-value memories and modern Hopfield retrieval, and shows when a kernel makes it convex by construction.Thu, 04 Jun 2026 19:00:00 GMThttps://tahabouhsine.com/welch-bound-jax-analysis/https://tahabouhsine.com/welch-bound-jax-analysis/A runnable companion to the Welch-bound latent-space post: generate GIFs and implement the JAX metrics that tell you whether embeddings are collapsing, wasting rank, forming a simplex, or pressing against the Welch floor.Tue, 02 Jun 2026 19:00:00 GMThttps://tahabouhsine.com/welch-bound-good-latent-space/https://tahabouhsine.com/welch-bound-good-latent-space/The hidden codebook inside representation learning: why collapse happens, why opposition is a trap, why class means form a simplex, and why the Welch bound sets the best geometry when too many concepts share too few dimensions.Tue, 02 Jun 2026 00:00:00 GMThttps://tahabouhsine.com/linear-attention-jax-flax-nnx/https://tahabouhsine.com/linear-attention-jax-flax-nnx/A runnable companion to Cheap Attention: implement positive-feature linear attention in JAX and Flax NNX, watch the all-pairs ledger turn into a shared feature state, and see exactly where the N×N matrix disappears.Sun, 31 May 2026 19:30:00 GMThttps://tahabouhsine.com/cheap-attention-is-linear-attention/https://tahabouhsine.com/cheap-attention-is-linear-attention/A 128K-token context creates billions of pairwise questions per attention head. But the N×N matrix is not the essence of attention; it is the receipt for an infinite feature map we never wrote down. Approximate that feature map with random features, reassociate the sum, and softmax attention becomes linear-time kernel attention. The whole argument is built from live in-browser visualizations.Sun, 31 May 2026 00:00:00 GMThttps://tahabouhsine.com/organizing-randomness-jax/https://tahabouhsine.com/organizing-randomness-jax/A block-by-block JAX + Optax implementation of six contrastive losses, each watched as a real animated GIF turning random 2D points into organized embeddings. The runnable companion to "Untangling the Moons."Tue, 26 May 2026 19:00:00 GMThttps://tahabouhsine.com/untangling-the-moons/https://tahabouhsine.com/untangling-the-moons/Eight contrastive losses, twenty years of history, one interactive playground. Watch pair, triplet, InfoNCE, CLIP, SupCon, SigLIP, alignment+uniformity, and cosine→0 organize 2D points, and see which ones know when to stop.Tue, 26 May 2026 00:00:00 GMThttps://tahabouhsine.com/what-an-mlp-knows/https://tahabouhsine.com/what-an-mlp-knows/The transformer MLP is illegible because its primitive does not carry a kernel. Give it one and the four objects that make attention legible follow for free, for the whole network.Thu, 21 May 2026 00:00:00 GMThttps://tahabouhsine.com/attention-is-kernel-jax-flax-nnx/https://tahabouhsine.com/attention-is-kernel-jax-flax-nnx/A runnable companion to Attention is Explainable Because it is a Kernel: build scaled dot-product attention from scratch in Flax NNX, prove in code that it is exactly a Nadaraya–Watson kernel smoother, watch the separate q/k projections break positive-definiteness numerically, swap the exp-dot-product kernel for Gaussian, Yat, and linear kernels to see which keep the weights a convex partition of unity, read the temperature as a kernel bandwidth, and train a single head end-to-end to route to a marked token.Thu, 14 May 2026 17:00:00 GMThttps://tahabouhsine.com/attention-is-a-kernel/https://tahabouhsine.com/attention-is-a-kernel/Self-attention in transformers is a Nadaraya–Watson kernel smoother. That fact, and not "we visualize the matrix", is why attention heads are readable while MLPs are not.Thu, 14 May 2026 00:00:00 GMThttps://tahabouhsine.com/not-all-infinities-are-equal/https://tahabouhsine.com/not-all-infinities-are-equal/The singularity structure of cross-entropy is asymmetric, and that asymmetry explains LLM hallucination, the CLIP modality gap, and why contrastive losses need 32K batches.Mon, 23 Feb 2026 00:00:00 GMThttps://tahabouhsine.com/opposite-is-not-different/https://tahabouhsine.com/opposite-is-not-different/Maximum difference between two unit vectors is orthogonality (cos = 0), not opposition (cos = −1). CLIP, InfoNCE, and SimCLR have been optimizing for the wrong target for years.Sun, 22 Feb 2026 00:00:00 GMThttps://tahabouhsine.com/activations-are-bad-for-geometry/https://tahabouhsine.com/activations-are-bad-for-geometry/ReLU, GELU, and friends factor into a layer's Jacobian as a diagonal modulation that wrecks the geometry of the data manifold. Why pointwise activations are a representational bug.Fri, 20 Feb 2026 00:00:00 GMT