Dynamical Systems
4 long-form posts on Dynamical Systems: machine-learning research by Taha Bouhsine, each built around live, in-browser interactive visualizations.
-
A Velocity Ledger for Transformers, in JAX/Flax NNX
A runnable companion: the pre-norm Transformer block as a forward-Euler step, then the residual-stream velocity ledger as one line of Flax NNX state (mu = 0 recovers plain), the ngpt-lite retraction variant, best-val early-stopped training, and the depth telemetry (path length and turning angle per sub-update). Four parameter-matched char-level GPTs that tie on quality and split on dynamics: the ledger's residual-stream path is a third as long and half as sharp.
-
Transformers With a Velocity Ledger
A pre-norm Transformer's residual stream is forward Euler: x += Attn(norm x); x += MLP(norm x). So D1's whole dictionary transfers, and the same question follows: does a velocity ledger in the residual stream buy in a Transformer what it bought in a ResNet? The answer splits. On quality, four variants tie. On dynamics, the ledger changes everything: the residual-stream path through depth gets dramatically shorter and straighter, reaching the same answer by a calmer journey. Same destination, gentler road.
-
Skip Connections With Inertia, in JAX/Flax NNX
A runnable companion: the residual block as a forward-Euler step, then the momentum residual network as a Flax NNX module with one extra state, a velocity the blocks write into. Train both on the rings task a first-order flow cannot separate exactly, watch the training crystallize, and run the trained network exactly backward until floating point, amplified by 1/mu per layer, steals the past.
-
Your Skip Connection Is Half of Newton
A residual block x + F(x) is one forward-Euler step: depth is time, the block is a velocity, position moves directly. That is half of Newtonian mechanics. A planet does not update position from force; force updates velocity, velocity updates position, and that split is why orbits are stable. So what does the missing half cost a deep network? We let the physics make three predictions about trained networks, then check all three live in the page. One of them comes back stranger than we wrote it.