series · 2 parts · in progress
Networks as Integrators
Numerical analysis as an architecture catalog: skip connections as an Euler step, momentum nets as half of Newton, and conservation laws as testable predictions about hidden states.
Start reading → Your Skip Connection Is Half of Newton- 01 Your Skip Connection Is Half of Newton A residual block x + F(x) is one forward-Euler step: depth is time, the block is a velocity, position moves directly. That is half of Newtonian mechanics. A planet does not update position from force; force updates velocity, velocity updates position, and that split is why orbits are stable. So what does the missing half cost a deep network? We let the physics make three predictions about trained networks, then check all three live in the page. One of them comes back stranger than we wrote it. JAX companion Skip Connections With Inertia, in JAX/Flax NNX
- 02 Transformers With a Velocity Ledger A pre-norm Transformer's residual stream is forward Euler: x += Attn(norm x); x += MLP(norm x). So D1's whole dictionary transfers, and the same question follows: does a velocity ledger in the residual stream buy in a Transformer what it bought in a ResNet? The answer splits. On quality, four variants tie. On dynamics, the ledger changes everything: the residual-stream path through depth gets dramatically shorter and straighter, reaching the same answer by a calmer journey. Same destination, gentler road. JAX companion A Velocity Ledger for Transformers, in JAX/Flax NNX
This series is still being written. New parts land in the archive and the RSS feed as they publish.