Transformers With a Velocity Ledger

A residual block $x + F(x)$ is one forward-Euler step through depth, and in a toy ResNet, carrying a second ledger alongside it was worth a planet’s worth of stability. If that reading is right, then every residual network is an integrator, and the most consequential residual network of the decade is the Transformer. Which is an invitation you cannot decline: walk up to a real language model and ask it the same question. Does giving the residual stream a velocity change what the model does?

Look at a pre-norm Transformer layer, the standard modern kind, and the correspondence is already sitting there in plain sight:

Two sub-updates per layer, each one a state plus an increment, each one a forward-Euler step on the residual stream $x$. The stream is position, the attention and MLP branches are the vector field, the same dictionary as before, now pointed at a two-hundred-dimensional stream of token representations instead of a width-2 plane. Which means the missing half of Newton is missing here too. Nothing accumulates. Each sub-update moves the stream directly, forgetting the direction of the last one.

So we did the obvious experiment. Take a small character-level GPT, hold everything fixed, and change only the residual-stream update: plain (the Euler layer above), ledger (the ResNet’s velocity stream, one line), ngpt-lite (a first-order variant that lives on a sphere, no velocity), and ngpt-ledger (the ledger and the sphere). Three seeds each, best-val early-stopped.

Here is the headline, up front, because the story is more interesting once you stop hoping for the wrong thing. On quality, the four tie. Best validation loss lands at 1.433 (plain), 1.435 (ledger), 1.420 (ngpt-lite), 1.456 (ngpt-ledger), a spread of 0.036 against a per-seed noise of about 0.005. The velocity ledger did not make the Transformer better. What it changed instead is the thing a validation number has no way of reporting, and the thing the physics predicted: the path. The residual stream reaches the same answer by a journey roughly a third as long and less than half as sharp. Same destination, calmer road. That is the finding, and the ResNet told us to expect it.

The dictionary still holds#

The move from a width-2 ResNet to a real Transformer is where an analogy like this usually cracks, and the attention block is the obvious place to look for the crack. There is no crack. Here is the correspondence as a table, since everything below reads off it:

The only genuinely new wrinkle is the norm: a pre-norm block reads a normalized copy of the stream, integrates the field there, and writes the raw increment back. That is a preconditioning of the field, not a change to the integrator; the update is still position-plus-increment, still Euler. This reading is not new either. Lu et al. (2019) read the whole Transformer as a multi-particle dynamical system and its layers as a Lie-Trotter splitting of an ODE; the ODE Transformer of Li et al. (2022) replaced the Euler step with a Runge-Kutta one and got a better integrator of the same field. Depth-as-time is an established lens on Transformers. What the ResNet reading adds is the specific missing piece, the velocity, and a specific prediction about what carrying it does to the trajectory.

And the prediction transfers verbatim. On the rings task the ledger’s signature was geometric rather than numerical: consecutive Euler steps forget each other and the path kinks, while a velocity that persists across steps makes the path coast, smooth and short. The plain ResNet turned 53° per layer; the momentum net turned 10° on a path of nearly the same length. So the question to bring to a Transformer is not whether the ledger raises accuracy, it is whether the ledger straightens the residual-stream trajectory. Hold that question. It is the one the data answers loudly.

This idea has precedent, and that is fine#

A velocity in the residual stream is not a new idea, and the place to say so is before any training curve appears. What follows is a clean, instrumented demonstration of a known effect, plus a deliberate contrast.

Where this sits in the literature. Closest is Momentum Streams for Optimizer-Inspired Transformers (2605.24425), which builds exactly this: an auxiliary velocity (and moment) stream running alongside the residual stream, with heavy-ball, Nesterov, and Adam-like variants. Their framing is ours: the velocity stream turns the first-order residual map into a second-order recurrence in depth, which is the ResNet result restated inside a Transformer. YuriiFormer (2601.23236) reads GPT layers as gradient descent on a token energy and adds Nesterov/Polyak momentum in the representation at no extra attention or MLP evaluations, the same acceleration from a different door. The Momentum Transformer (2208.00579) puts momentum inside linear attention rather than the residual stream. The older ODE lineage (Lu et al., 2019; ODE Transformer, 2203.09176) is the depth-as-dynamics reading we are standing on. And nGPT (2410.01131) is the deliberate contrast: a first-order variable-metric optimizer on the hypersphere, with normalized displacements and a retraction step but no velocity at all. Our ngpt-lite is a simplified nGPT (the sphere and per-dimension step scales, without the weight and logit normalization); it is the first-order-on-a-sphere corner that makes the ledger’s effect legible by contrast.

So the experiment is a measurement, not a claim of invention: the depth dynamics the ResNet predicted are real and visible in a Transformer, and here is what the velocity does and does not do, with the nGPT sphere as a control.

The scope, before the claims#

The result concerns dynamics, so the measured object comes first.

What this experiment is and is not. It is one small character-level GPT (6 layers, 192-wide, 4 heads) on one corpus (the complete works of Shakespeare, character level), trained for 12000 steps with dropout, best-val early-stopped, three seeds. It is an illustration of a dynamical effect at fixed parameter count, not a scaling claim, not a recommendation, and not evidence that a velocity ledger helps at any size. All four variants are matched to within 0.1% of parameters (the ledger adds zero parameters; it is a depth-state, not a weight). The depth telemetry is measured on a fixed held-out probe batch. Read every number below as “on this model, this is what the ledger does to the trajectory,” and nothing wider.

The tie on quality#

Every hope you might bring to a new architecture knob dies on the first figure, and it is better to look straight at it. Best validation loss, three seeds, four variants, on a zoomed axis so the whole spread is visible:

One more number, so that an earlier version of this story is not misremembered: the over-training blow-up is gone. In an earlier, smaller-corpus run the ledger and plain variants over-trained at very different rates, which looked like a robustness signal. With dropout and a five-times-larger corpus, every variant’s final val loss sits essentially on top of its best (blow-up ratio 1.00 across the board), so there is no hidden robustness win either. Just a clean four-way tie on quality, which is why the interesting part is not on this axis at all.

Where the signal actually lives#

If the four models end up equally good, are they doing the same thing to get there? Not remotely. The whole payoff of the depth-as-time reading is that it hands us an axis the scoreboard does not have: the geometry of the path the residual stream takes from the embedding to the readout. Measure that path on the probe batch and the four variants separate cleanly.

Two things to read carefully here, because they are the crux. First, the ledger path is dramatically shorter: the residual stream does not need to travel 128 units to land where plain lands; carrying a velocity gets it to an equally-good answer in 48. Second, it is dramatically straighter: 29 degrees of bend per step against plain’s 75. That is the ResNet signature, arriving intact. There the momentum net made the same-length journey without the kinks; here the momentum stream makes a shorter journey and still without the kinks. The velocity is doing the same job it did on the planet: accumulating direction across steps so the state stops re-deciding where to go at every layer.

The ngpt contrast sharpens it. ngpt-lite, first-order on the sphere, bends the most of all, 98 degrees a step, because retraction to the sphere after every sub-update keeps re-orienting the stream; its path is short only because the sphere caps how far it can wander. Add the ledger to it and the bend drops to 47 and the path to 32: the velocity smooths even the sphere’s re-orientations. Momentum straightens the path whether the geometry is flat or spherical.

You can see the trajectory itself. The residual stream is 192-dimensional, so this is a 2D shadow, but it is a faithful shadow: every segment you see has the real measured length, and every bend has the real measured angle. The verdict your eye reads, shorter and straighter, is the model’s measured verdict.

Why the ledger’s path is short#

The shortening has a mechanical source, and it fits on one line, the same line the ResNet used:

with $\mu = 0.9$, $h = 1$ fixed, one velocity stream shared across both sub-updates of every layer, zero-initialized at the embedding. Set $\mu = 0$ and it collapses exactly to the plain layer $x \mathrel{+}= F(\mathrm{norm}\,x)$, so the ledger is one dial from Euler to heavy-ball, at identical parameter count.

The $(1-\mu)$ factor is the whole story. A plain layer writes the full block output $F$ into the stream; the ledger writes a running average of $F$, damped by $0.9$. Early in depth, before the average has filled, each step is tiny; the velocity accumulates gradient across layers and only opens up near the readout. Watch the actual step sizes:

The plain net’s step size balloons toward the readout, dumping a 25-unit increment in the final sub-update, which is where much of its 128-unit path and its stream-norm blow-up to 72 come from. The ledger’s steps stay small and even, because the $(1-\mu)$ damping refuses to let any single block output move the state far, and the velocity that funds each step is a smoothed accumulation, not a raw output. That is the whole shortening: the ledger integrates the field instead of chasing it.

The path straightens as training proceeds#

Is the straight path there from initialization, or does the model learn to keep it short? The telemetry was dumped at four checkpoints (init, ten percent, fifty percent, and the end of training), so we can watch the gap open.

At initialization everything is short, because an untrained field is small. Training is what pushes the plain net’s path out: as the plain net learns to separate characters, its per-layer steps grow, its stream-norm climbs, and its path lengthens toward 128 and stays there. The ledger, over the same training, holds its path near 48. The straightness is a trained property of carrying a velocity, not a coincidence of the starting point, and it survives to the checkpoint that scores best. Same best-val, and one path a third the length of the other, at every point past the first tenth of training.

Same answer, gentler journey#

Not accuracy, then. On this model the four variants tie on best validation loss to within seed noise, the over-training blow-up is gone, and the ledger is not a quality upgrade. That is the headline, and it is a tie.

What it did instead is what a second ledger was always supposed to do: integrate calmly. The residual stream reaches an equally-good answer along a path that is roughly a third as long and less than half as sharp, and it does so whether the geometry is flat (ledger vs plain) or spherical (ngpt-ledger vs ngpt-lite). The velocity accumulates direction across depth, so the stream stops re-deciding where to go at every sub-update and starts coasting, the way a thing with inertia moves. The plain net gets to the same place by lurching, a fresh 25-unit jump at the readout; the ledger glides.

Whether a shorter, straighter residual-stream path is worth anything is a genuinely open question this experiment does not answer. It might matter for very deep stacks, where a first-order flow’s accumulated wander could be a real cost; it might matter for interpretability, since a straight path is easier to read; it might not matter at all at this scale, which is why the word here is tie. What the data does settle is that the prediction transferred and held: a velocity in the residual stream is a smoother integrator of the same field, and the calm it produces shows up in every telemetry column we dumped. Half of Newton runs the Transformer too, and the other half, once again, arrives as a straighter road to the same place.

Every number in this post comes from one run in JAX and Flax NNX: four parameter-matched char-level GPTs, three seeds, best-val early-stopped, with depth telemetry on a fixed probe batch. The runnable companion builds all four residual-stream updates in Flax NNX and renders the depth telemetry from the same bundle.

@misc{bouhsine2026transformerswithavelocityledger,
  author       = {Bouhsine, Taha},
  title        = {Transformers With a Velocity Ledger},
  year         = {2026},
  month        = {jul},
  howpublished = {\url{https://tahabouhsine.com/blog/transformers-with-a-velocity-ledger/}},
  note         = {Blog post, Records of the !mmortal Data Scientist}
}