You Don't Even Have to Train the Features

The last post split a network into features and a classifier, trained the features, and built the classifier by hand. It ended on a greedy question: if the classifier comes for free, how much of the features can you get without training either? This post answers it. Build the features by hand too, the way computer vision built them for decades, edges and corners and nothing learned, feed them to the same constructed head, and the whole network, with no training anywhere, matches the trained backbone from the last post point for point. Every number is from a real run; the script is scripts/handbuilt_vision.py.

The part everyone says you cannot skip#

You can skip the classifier. The previous posts made that case three times over: a neuron is a picture, a network is a list of those pictures, and you can place them by hand instead of learning them. But all of that sat on top of features, and the features came from a trained backbone. That backbone is supposed to be the irreducible core, the part deep learning exists to provide. The whole story of the last fifteen years is that you cannot hand-design good features, you have to learn them.

People did hand-design them, though, and it worked. Before the deep-learning turn, the best vision systems ran on features a person wrote down: oriented-edge histograms, corner and blob detectors, gradients pooled over a grid. HOG and SIFT did not learn anything; they measured the image in ways an engineer chose, and those measurements fed a plain classifier. So the greedy question has a concrete version. Take features of exactly that kind, hand-built and frozen, hand them to the constructed head from the last post, and see how far a network with nothing trained anywhere can actually get.

A feature you can name#

What is a feature, underneath? The last post called it a learned measurement, a number a backbone is trained to compute. A hand-built feature is that same thing with the word “learned” struck out: a measurement you choose, and because you chose it, you can name it. Here is the whole bank.

Six of them look for an edge at a fixed angle; the seventh looks for a corner, a place where two strong edge directions meet at once. That is it. No filter here was tuned to Fashion-MNIST or to anything else. They are the generic structure of natural images, written down.

And written down literally, because the whole bank is a closed form. Two Sobel kernels give the image gradient, $g_x = S_x * I$ and $g_y = S_y * I$, and the gradient carries the only two facts any of these detectors reads, how strong an edge is and which way it points:

An oriented-edge channel is then the edge energy weighted by how close the edge angle sits to that channel’s preferred direction $\theta_b = b\,\pi/6$. With a triangular window of half-width $\Delta = \pi/6$, the circular distance $|\cdot|_\pi$, and $[\,\cdot\,]_+ = \max(0, \cdot)$,

and the corner channel fires where both gradient directions are strong at once, $e_{\text{corner}}(p) = |g_x(p)|\,|g_y(p)|$. Seven maps, not a parameter among them. Run an image through and it stops being a grid of pixels and becomes a stack of these named measurements, one per detector, each lit where its kind of edge is present.

Where do the dimensions come from?#

Seven full-resolution maps per image is more than a classifier wants, so each one gets boiled down, and how it boils down is where the dimensions are born. Cut the image into a grid of patches and, in each patch, record how much of each detector fired. That is the whole feature vector: one number for every (detector, patch) pair. With seven detectors and a seven-by-seven grid that is 343 numbers, and the move that matters is what each one means.

How much machinery does that boiling-down take? One line. Average each channel $d$ over each patch $P_{ij}$ of the $7\times 7$ grid, stack the results, and divide by the length:

That final normalisation is the one concession to robustness, the HOG trick that sends a bright garment and a dim one to the same place. Set $\phi$ beside a trained backbone $g_\theta$ and the whole post sits in a single comparison: $g_\theta$ has millions of parameters $\theta$ that gradient descent went looking for, while $\phi$ has exactly none. It is a fixed function, written once and never moved.

That is also the difference between a constructed representation and a learned one, made literal. A trained backbone turns an image into a vector that separates the classes too, but ask what its tenth coordinate measures and there is no answer in words; it is whatever gradient descent found useful. Here the tenth coordinate is “thirty-degree edge in patch four,” and you assigned it. You decided, in advance, which information lives on which axis. The representation is legible before the data ever arrives, because you built it to be.

Hand it to the built head#

So what reads these 343 named numbers? Nothing new, and that is the point. The classifier is the constructed Yat head from the editing post, the kernel $k(z, \mu_u)$ that post wrote out in full: twenty prototypes per class, each $\mu_u$ a k-means centroid of that class’s training features. It is the same softened inverse-square well that post dropped masses into, a denominator that pulls hardest where the feature lands nearest a prototype, except the space it pulls in is now hand-built features instead of pixels. Each class is represented by its nearest prototype, and the strongest class wins:

Write the network out in full and there is nowhere for training to hide. It is $\hat y \circ \phi$, a fixed feature map composed with a parameter-free vote, and no gradient step touches the detectors, the pooling, the prototypes, or the readout. You can follow a single prediction the whole way down and never meet a number that was learned.

Over the whole test set, this fully hand-built network reaches 83.3%. For reference, the same constructed head on raw pixels managed 79%, and a single centroid per class, the simplest head imaginable, already gets 75.6%. The hand-built features buy real accuracy over pixels, and they do it with detectors a person could have written in 2004.

Run it yourself#

Eighty-three percent from edges and a vote is the kind of number you should not take on faith, so do not. Draw a garment in the pad below, or load a real one, and the entire network runs right here in the page: the seven detectors fire on your strokes, the patches pool into 343 numbers, the kernel scores your drawing against the stored prototypes, and a class wins. There is no trained model loaded anywhere to fall back on, only the detectors and the vote. The exact pipeline you are watching is scripts/handbuilt_vision.py in the repo, line for line, so you can run the whole experiment yourself and get the same 83.3%.

A crude drawing will not classify as cleanly as a real photo, which is the point: the detectors are generic, not tuned to your handwriting or to anything else. But load a real garment and it lands, every time, with the same edges and the same vote you could compute by hand.

Where the line finally falls#

So how does building the features by hand compare to learning them? A trained convolutional backbone, with the same constructed head on top, scored 83.2% in the last post. The hand-built features match it, point for point. Train the head too, end to end, and you reach 85.7%. (Both learned-backbone rungs are from the reference run in scripts/construct_vs_optimize.py, the same run the last post read.) The whole series fits in one chart.

Read it as a ladder and the boundary is finally visible. Construction reaches all the way up to 83.3%, and everything below that line is a network you can read end to end, no weights, no mystery, every dimension named and every prototype a picture. The last 2.4 points, the climb from 83.3 to 85.7, are the only thing training buys, and they are real: gradient descent finds features tuned to this data, the particular ways a sandal differs from a sneaker, that no one would think to hand-write. That is what learning is for. Not the bulk of the accuracy, which generic edges and a nearest-prototype vote already deliver, but the last slice that comes from fitting the detectors to the problem instead of to images in general.

What this says about the whole thing#

What was a network, then, all along? Read across these posts and the arc is a single deflation, in a good sense. A network looked like an indivisible black box; it turned out to be a feature map with a classifier on top, and we took it apart one piece at a time. The classifier is a list of pictures you place. The features are measurements, and the measurements can be ones you name and write down. Put both together and “a deep network” is, at its readable core, a bank of detectors feeding a nearest-prototype vote, and every part of that core is something you can construct rather than optimize, though only one feature layer deep so far: how far down a truly deep backbone can be built, layer by layer, is still an open question.

This is also just the history of computer vision, run forward and then read backward. The field spent decades hand-building features, switched to learning them, and got a few points and a great deal of opacity in trade. Looking back through the constructed network, you can see exactly what the trade was: learning bought the last 2.4 points and charged you the ability to say what any dimension means. Whether that is a good deal depends on what you need. If you need the accuracy, train it. If you need to know what the network is doing, you can now build almost all of it by hand and read every line.

And the deflation leaves one part of the black box standing. The classifier turned out to be placement, and the features turned out to be measurements; but real networks also stack layer upon layer, and nothing in this series has yet said what all that depth is for. That question is next.

Hand-built oriented-gradient features are the HOG idea from Dalal and Triggs (2005), in the lineage of SIFT (Lowe, 2004); the prototype head (“this looks like that”) is from Chen et al. (2019); Fashion-MNIST from Xiao et al. (2017); the Yat kernel from Bouhsine (2026). This closes the thread opened in You Only Have to Train the Features.