Records of the !mmortal Data Scientist

series · 12 parts · in progress

The Prototype Network

Replace the activation with a finite, positive-definite kernel and a network becomes a list of prototypes you can read, edit by hand, and finally collapse into a single fixed-point operator.

Start reading → What a Finite Kernel Buys an MLP
  1. 01 What a Finite Kernel Buys an MLP Replace the activation function with a finite, explicit, positive-definite kernel, the Yat kernel, and an MLP stops being a stack of linear maps glued by a nonlinearity. It becomes a kernel machine, with locality, attribution, geometry, capacity control, and a feature map you can write down. JAX companion The Yat-Kernel MLP in JAX/Flax NNX
  2. 02 Your Neuron Is a Direction. It Should Be a Picture. Why should a neuron store a direction when it could store a thing? A direction is not a referent you can point at, which is why MLPs are opaque. Put the Yat kernel where the activation was, train on Fashion-MNIST, and every neuron becomes a prototype that lives in pixel space, literally a picture, so the network reads its own predictions: this looks like that, no saliency method required. JAX companion Your Neuron Is a Picture, in JAX/Flax NNX
  3. 03 Your Network Is a List of Pictures. You Can Edit It. If a neuron is a labelled picture, a classifier is a list of them, and a list is something you edit. Add a class to a trained-free Yat-kernel network by placing twenty pictures, and it recognizes that class at 95% with zero gradient steps. Delete a class by removing its pictures, and it is forgotten exactly, the other classes untouched. Class-incremental learning with no penalty and machine unlearning that is instant and exact, both falling out of the architecture rather than bolted on. JAX companion Editing a Network by Hand, in JAX/Flax NNX
  4. 04 You Only Have to Train the Features Leave a convolutional network's weights at their random starting values and build a Yat head on its features by hand: the trained head on that random backbone sorts at chance while the constructed one reaches 74%. On a properly trained backbone the constructed head reaches 83.2% against 85.7% for the trained one. The accuracy lives in the representation; the classifier, and its edits, are furniture you place. This maps the boundary between what you must optimize and what you can construct. JAX companion Constructing the Head on Learned Features, in JAX/Flax NNX
  5. 05 You Don't Even Have to Train the Features The last post trained a backbone and built the classifier by hand. This one builds the features by hand too: oriented-edge and corner detectors pooled over a grid of patches, the way computer vision worked for decades. Feed those to the same constructed Yat head and, with nothing trained anywhere, it matches the trained backbone on Fashion-MNIST point for point, within a couple of points of a fully trained network. The whole network is hand-built and readable end to end. JAX companion The Hand-Built Network, in JAX/Flax NNX
  6. 06 How Far Down Can You Build? One hand-built feature layer matched a trained backbone at 83.3% on Fashion-MNIST, and real networks are deep. Conveniently, the recipe for a second layer has been on the shelf for half a century: vision science says edges assemble into junctions, continuations, bends and stripes. This post takes the recipe down and follows it, builds layer 2 entirely by hand with every dimension still nameable in one sentence, and measures exactly where construction stops, and why. JAX companion Building the Second Layer by Hand, in JAX/Flax NNX
  7. 07 When 80% Should Mean 80% A network hands you a probability with every answer, and the number is the part you act on. So when this series' bounded, self-explaining kernel network says 80%, is that a measurement or a mood? Five posts of evidence say it should be the honest one. This post puts that reputation through a lie-detector test, reliability diagrams, expected calibration error and temperature scaling against a matched ReLU MLP on Fashion-MNIST, and what the test found is the post. JAX companion Calibrating a Bounded Net, in JAX/Flax NNX
  8. 08 A Risk Model That Names Its Reasons A survival model tells an oncologist a patient is high-risk, and she has to act on the number without being able to ask why. What would it take for the risk score to name its reasons? We build a Yat-kernel DeepSurv on breast-cancer survival, match a standard DeepSurv on concordance, and get a risk score that decomposes exactly into the prototype patients this one resembles, a model you can read, audit, and edit. JAX companion A White-Box DeepSurv, in JAX/Flax NNX
  9. 09 The White-Box Survival Model on Trial A classical kernel machine is beautiful and needs a solve that will not minibatch or compose. A deep net trains on anything and its risk score is a fog. What if one thing had the training of a net and the theory of a kernel? We put a deep Yat-kernel survival model on trial across five real datasets against Cox, penalized Cox, and Random Survival Forest: it trains with plain gradient descent, lands in the pack on the concordance index, and, because its units are genuine Mercer kernels, inherits exact attribution, calibration, editing, and bounded out-of-distribution response the others cannot give. Where it costs a point or two, we show exactly where. JAX companion Running the Survival Trial, in JAX/Flax NNX
  10. 10 Your Network Is a Stack of Layers. It Could Be a Fixed Point. A deep network makes you choose its depth before you have seen the problem, and gives every layer its own weights. Share one Yat-kernel operator across all of them instead, and the stack collapses into a single equation: the answer is the fixed point the state settles into. Training makes that operator a contraction, so the settling point is unique and reached from anywhere, the network decides its own depth per input, and the same twenty-four prototypes describe the computation at every step. 98.2% on two moons from 1700 parameters shared across all depth. JAX companion A Network That Is a Fixed Point, in JAX/Flax NNX
  11. 11 Edit One Operator, Edit Every Depth One post taught and forgot classes by editing rows of a Yat network, with proofs that nothing else moved. Another melted the stack of layers into a single operator iterated to a fixed point. This is the collision. Every one of those editing proofs rested on a pasted row entering the score once, as one term in one sum, and in an equilibrium network there is no once: whatever you paste is applied at every depth and fed back into its own input, and every fixed point is free to drift. So did melting the stack melt the editability? This post pastes, deletes, and measures: every guarantee that survives is either proved inside the recursion or measured against the real run, fixed point by fixed point. JAX companion Editing a Deep Equilibrium Network, in JAX/Flax NNX
  12. 12 One Kernel, Fitted Twice Kernel methods gave us the theory everyone still wants back, and the field abandoned them over one procedure: the O(n³) solve over an n by n Gram matrix, which cannot minibatch, cannot scale, and cannot sit under other layers. So we took one Mercer kernel and fitted it twice: once by the classical exact solve, once by plain gradient descent on a bank of prototypes. The two machines agree, to a correlation of 0.95, and then the descended one walks through three walls the solved one dies at: a measured memory wall at sixteen thousand rows, a half-million-row dataset the solve cannot touch, and an end-to-end network the solve cannot be. JAX companion Solving It and Descending It, in JAX/Flax NNX

This series is still being written. New parts land in the archive and the RSS feed as they publish.