How Far Down Can You Build?

The last post built a network with nothing trained anywhere, one hand-designed feature layer feeding a constructed head, and it matched a trained backbone at 83.3% on Fashion-MNIST. It ended on the question it could not dodge: real networks are deep, and construction had only reached one layer down. So this post does the obvious next thing. It builds a second feature layer by hand, junctions, continuations, bends, stripes, every dimension still nameable in one sentence, and measures what that buys. The answer is the finding. Every number is from a real run; the script is scripts/handbuilt_depth.py.

The recipe is already on the shelf#

Twice now, the greedy move has paid. The classifier was supposed to need training, and it turned out to be placement. The features were supposed to need training, and one hand-built layer of them matched the trained backbone point for point. Each time, the part everyone called irreducible came apart into measurements a person could write down. So the greedy logic says: keep going. The fully trained network sits at 85.7%, the hand-built one at 83.3%, and between them, supposedly, lies the thing deep learning is actually for, depth, layers stacked on layers. Build one more layer by hand and see whether the gap closes.

And here is the tempting part: you would not even have to invent the second layer. Vision science wrote it down decades ago. Hubel and Wiesel found oriented-edge cells in the first cortical stage of vision, the exact detectors the last post built, and the stages after it respond to arrangements of edges: corners, junctions, curved contours. Marr’s primal sketch tells the same story from the computational side, edges first, then the tokens edges assemble into. The recipe for layer 2 has been on the shelf for half a century. This post takes it down and follows it.

One stake before we start. If the second layer climbs, construction eats into the last 2.4 points and the thesis of this series gets stronger. If it does not climb, that is not a failed experiment; it is the first real measurement of where the construction program stops, and why. Either result is worth having. The ladder, three sections from here, will pick one.

What can you say about an edge?#

A second layer has to be made of sentences about the first, so start with what the first layer says. Its whole vocabulary is seven words, six oriented edges and a corner, each measured at each patch of the image (the last post built them from Sobel gradients; we reuse that recipe unchanged). A second layer, then, should say the things you can only say by relating those words. And when you sit down to write such sentences, four kinds come out almost by themselves:

• A junction. Two different orientations firing in the same cell: “a 30° edge and a 90° edge meeting,” the corner of a collar, the crotch of a trouser.
• A continuation. The same orientation firing one cell along its own direction: “a 0° edge that keeps going,” the long clean side of a trouser leg rather than a fleck of noise.
• A bend. An edge whose continuation arrives rotated one step: “a 60° edge turning into a 90° edge,” the curve of a shoulder or a shoe’s toe.
• A stripe. The same orientation echoed two cells across, perpendicular to itself: “two parallel 90° edges,” the two sides of a strap or a sleeve.

Each of these is an and, and the cheapest faithful AND between two non-negative energies is a minimum. Write the layer-1 edge energies on a grid of cells as $e_b(p)$, with $u_b$ the unit step along orientation $b$‘s own direction, and the whole second layer is four lines:

Fifteen junction pairs, six continuations, six bends, six stripes: 33 named detectors, each one sentence long, computed from layer 1 the way layer 1 was computed from pixels. Watch two edge maps become a junction below; every map is computed live from the garment, the same arithmetic as the script.

Does depth cost the names?#

The first thing depth was supposed to take from us is legibility, so check that before checking accuracy. Layer 2 gets the other habit of depth too, a coarser grid: its maps are pooled over 4×4 regions instead of layer 1’s 7×7, the way real networks trade resolution for abstraction as they go up. That makes 33 detectors × 16 regions = 528 new dimensions, and every one of them is still a sentence: “a 30°+90° junction in the top-right region,” “a 0° continuation along the bottom.” Click around the whole vocabulary below.

So legibility survives the second layer completely. That was the obvious way for this experiment to fail, and it is not what happens. A constructed network two layers deep is exactly as readable as one layer deep; the sentences just get more interesting. Whatever wall exists between construction and depth, it is not made of opacity.

The second rung#

It is made of something stranger, and the ladder shows it. The setup could not be more favorable: 528 new dimensions of curated, century-old vision science, concatenated onto the 343 that already reached 83.3%, feeding the same constructed head as always, k-means prototypes and a nearest-prototype vote in the wells the editing post mapped. If depth-by-hand works at all, this number climbs toward the trained network’s 85.7.

It moves by −0.4. The full two-layer network scores 82.9%.

Maybe something specific is broken, so iterate like you would on any model, each redesign a theory about what went wrong. Maybe the weaker word types are diluting the strong ones: junctions alone score 83.5%, the best of the six, two tenths of a point above layer 1. Add continuations back: 82.8. Add bends: 83.0. Add stripes and you are back at the full vocabulary’s 82.9. Maybe min is too cruel an AND, throttling every junction to its weaker edge, so swap in a geometric mean: 83.2. Maybe one cell is too myopic for a shoulder-scale bend, so give every detector a two-cell reach: 81.8, the worst of the lot. Six designs, six theories about what was broken, and each came back refuted by a few tenths; the best of them buys +0.2 while most buy less than nothing. And the refutations share a shape: no theory rises, no theory matters, as if the head were shrugging at every new word offered to it. The second rung of the ladder is not lower than expected. It is flat.

Now the natural diagnosis, and why it is wrong. Flat rung, so the detectors must be junk? Run layer 2 alone: throw away the 343 edge dimensions entirely and classify garments from nothing but junctions, continuations, bends and stripes. It scores 78.8% (80.9% under the geometric-mean AND), within a fifth of a point of the raw-pixel network’s 79.0. Detectors that never see a raw edge recover almost the entire baseline. The words are real words; they carry genuine class signal.

They are just synonyms. Every layer-2 value is a fixed function of layer-1 values, $\min(e_{b_1}, e_{b_2})$ says nothing the pair $(e_{b_1}, e_{b_2})$ had not already said, and the constructed head, which reads whole feature vectors through a kernel, was already comparing garments on those coordinates jointly. Rewriting the same information in a richer vocabulary gives the head nothing to vote with that it did not have. (It even costs a little: the nearest-centroid control drops from 75.6 to 74.1 when the redundant dimensions pile on.) A trained second layer does not have this problem, because it is not restricted to re-describing; gradient descent can build combinations weighted precisely by what the first layer’s description misses.

Counting the wall#

But wait, the obvious objection goes, maybe we just picked the wrong 33 sentences. That objection is correct, and it is the finding. Count what “picking the right ones” would mean. Layer 1 has 7 words. A pairwise layer-2 detector is two channels at a relative offset, and within even a one-cell neighborhood there are 224 distinct nameable types. Go to three-way combinations, the kind a real second layer of a CNN routinely represents, and the space is roughly 4,630 types before you have chosen pooling, reach, or the form of the AND. We curated 33 of the 224 using the best prior knowledge our species has about early vision, and moved the needle +0.2 at best. The vocabulary explodes; the accuracy per name collapses.

And that recount changes what the wall is made of. It is not that a second layer cannot be named; we named 33 dimensions and could name all 224 without breaking a sweat, and each would still be one plain sentence. It is that at depth two, naming and helping come apart. At layer 1 the two coincided: there are only a handful of local, first-order things an image measurement can be, orientation and cornerness roughly exhaust them, so human priors could curate the vocabulary completely, and did, decades ago. One layer up, the design space has already outrun curation, and picking the few combinations that add signal, jointly, against what the layer below already provides, is a search problem over thousands of candidates coupled through the data. There is a name for searching a huge combination space with the data as the judge. It is called training.

What learning is for, said precisely#

So the series’ central trade can finally be stated with numbers on both sides. The last post said learning buys the final 2.4 points and charged the ability to say what a dimension means; this post says where those points live: not in layer 1, which a person can build completely and match, and not in the classifier, which has been placement since the editing post, but in the selection of deep combinations, which of the thousands of true sentences about edges are worth measuring. And that recasts what this series has been doing all along. Construction is not a rival training procedure that fell 2.4 points short; it is a measurement instrument. Run it up a network and it reports, layer by layer, how much of the machine is generic structure a person could have written (all of the classifier, all of the first feature layer, 83.3 of the 85.7 points) and how much is data-selected combination (the rest). One layer down, the instrument reads: this far by hand, no further. Though if selection really is the wall, a quieter thought opens past it: maybe depth was never a stack to build at all, but a loop to run. For now, the greedy question that drove three posts, how much do you get without training, has its complete answer, and the answer ends with a wall you can count.

Oriented-edge first stages are from Hubel and Wiesel (1962); the edges-then-tokens grammar is Marr’s primal sketch (Vision, 1982); the layer-1 detectors follow the HOG lineage of Dalal and Triggs (2005); Fashion-MNIST from Xiao et al. (2017); the Yat kernel from Bouhsine (2026). This answers the question left open in You Don’t Even Have to Train the Features.