Phase 07 / Transformers & language models
A Transformer block and mixed precision
Revision v0.18.1 · Updated since your last visit. Review the changes; your progress is saved.
Lesson · about 90 minutes · CPU practice
Build a residual attention block and compare numerical behavior across precision choices.
Phase study guide: preparation, milestones and project evidence
Before you start
Before this: Masking, tokenization, and sequence packing
Hardware: CPU
Setup help: Python, notebook, or Cloud TPU VM
Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions require separate checks. Hardware receipts apply only to their recorded source, environment, hardware, and cases; they do not qualify every lesson or TPU configuration. See the TPU performance and hardware evidence guide. The optional Cloud TPU specialization explains target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.
Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txtpy -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txtAfter activation, run the complete worked example from that same folder:
python phases/07-transformers/03-a-transformer-block-and-mixed-precision/code/main.py.\.venv\Scripts\python.exe phases/07-transformers/03-a-transformer-block-and-mixed-precision/code/main.pyFor an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.
Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.
For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:
python -m pip install jupyterlab
python -m jupyter lab.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter labSelect the downloaded transformers-03.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.
The worked reference for this lesson is CPU-validated. Some references explicitly configure logical CPU devices; setting JAX_PLATFORMS does not adapt them to a TPU. Follow the execution matrix and reviewed multi-host example before attempting a target run.
For accelerator preparation, study TPU setup, observed runtime receipts, checkpointed jobs and precision and retained traces. Current receipts cover bounded v5e1 launch, resume, and precision checks; four single-device Pallas cases; and a four-device v5e collective fixture. Each receipt is limited to its recorded source, environment, cases, and hardware. It does not qualify all lessons or TPU configurations. Cloud provisioning is optional for CPU learning.
Check: Output shape ; finite values, zero-projection identity, compiled/eager agreement.
Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.
What you will be able to do
- Follow the two residual paths
- Normalize features rather than examples
- Test causal independence and differentiation
- Separate storage precision from sensitive arithmetic
The problem
An attention head mixes information across positions. What else makes it a Transformer block? We’ll add normalization, a small feature-processing network and two residual connections. Then we’ll inspect gradients and compare a lower-precision version. This is a deliberately small CPU teaching block, not a production architecture.
The idea
A Transformer block adds learned corrections to token representations. This lesson uses a single attention head, normalization before each branch, and two residual additions. Follow the implemented dimensions rather than assuming all Transformer architectures are interchangeable.
Follow the residual stream through the actual block
A transformer block must both mix context and preserve a route for the original representation. Here four token rows each have eight features. Attention forms four-by-four scores, mixes eligible value rows, and projects the result back to width eight so it can be added to the input. The feed-forward branch acts on each token independently, expanding 8 to 16 features and back to 8 before the second residual addition.
The recorded block returns shape . Its heatmap shows output minus input, not the raw token representation. The nearly repeated correction has a fixture-specific cause: input rows differ by constant feature offsets, which per-token normalization removes. Change one feature of one row to test whether that pattern persists.
The actual single-head pre-LN block
Predict: What should happen when every learned projection is zero?
Follow the downward arrows: is normalized to form Q/K/V, Q/K scores become a causal four-by-four weight matrix, and the weighted values are projected back to a four-by-eight correction. The curved left bypass carries directly to the first residual addition. The next path normalizes that sum, expands each token from 8 to 16 features, applies GELU, contracts back to 8, and adds a second correction; the curved right bypass carries the intermediate residual to this addition. Every arrow carries four token rows, while the feature width changes only inside the MLP. The side arrow labeled routes values into the weighted-value step.
Follow the two residual paths
Read the diagram down the center for the main computation: normalize, form Q/K/V, apply causal attention, project a correction, and add it to the carried input. The next normalization and MLP produce another correction before the second addition. The curved arrows are skip paths carrying the original to the first add and the intermediate residual to the second; each add preserves shape .
If all six projection matrices are zero, both learned branches contribute zero and the output must equal . This makes a useful prediction before inspecting the implementation.
x → norm → Q/K/V → causal attention → output projection
└────────────────────────────── add → residual
residual → norm → expansion → GELU → contraction
└────────────────────────────────── add → outputNormalize features rather than examples
Layer normalization computes mean and variance across the feature axis of each token row, then divides by . For , the mean is 2, variance is 1, and the normalized values are approximately . A shared finite offset across one row disappears within tolerance, while a change to one feature need not.
This minimal function has no learned scale or bias and does not aggregate across batch or sequence. Its output shape matches its input, so the following residual addition stays well-formed.
Test causal independence and differentiation
The causal mask gives a direct independence test: heavily perturb the final token and the first three output rows should remain fixed. Normalization and the MLP work per token; the mask prevents earlier queries from reading the last key. Then check that the gradient pytree has the same leaf shapes as the parameters and finite values for this input.
Finite gradients alone do not validate the block. Pair them with the zero-projection identity case and eager/compiled agreement; interpret gradient magnitude in light of the loss reduction.
Separate storage precision from sensitive arithmetic
Casting every input and parameter to bfloat16 is a controlled low-precision comparison, not a full mixed-precision training recipe. The second policy keeps normalization and score softmax reductions in float32, then casts results back for lower-precision projections. Compare both against the float32 output on this fixed input.
The recorded maximum absolute error is about for each policy. That shared value does not establish equal behavior on other inputs or workloads. Inspect which operations retain precision and evaluate an error tolerance against the intended task before choosing a policy.
Configure depth without changing the block contract
This worked function remains one eight-wide block. The separate foundations model config varies width, depth, head count, MLP width, context, dropout and tied-output policy; it checks the parameter-count formula against the actual parameter tree. Its small CPU pilot uses a two-layer FP32 decoder, finite padding rows excluded from loss, and per-block projected keys and values for cached decoding.
A formula for a larger model is a sizing estimate, not an allocated model or a training result. Inspect the configuration and evidence together before transferring conclusions from this four-token example.
A residual block adds a correction to its input
Predict before looking: Does a residual connection force the output to equal the input?
Read the figure
Rows are the four token positions; columns are features 0–7. Each cell is the signed correction , with the diverging colors showing positive versus negative values and near-white showing a small change. Feature 3 increases by about , while feature 7 decreases by about ; those values repeat across rows to floating-point precision.
Connect it to the computation
The code subtracts the input from the block output for this plot, so a large color means a large correction, not a large final activation. The repeated rows follow from the chosen constant-offset inputs and per-token normalization; they are not a general property of attention. Predict how changing feature 3 in only token 1 will affect its row and later causal rows, then compare the new correction map.
Figure data and execution details
Open full-size figureDownload data and provenance
The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.
Your build: Create a separate main.py in your practice folder. Append each block in order and run python main.py in the lesson environment. The downloaded script is the complete reference, so keep it separate from your attempt.
1 Prepare the experiment
In the CPU environment from setup, create main.py and add this block. Write down the dimensions and the known result before continuing.
# Step 1 — 1. Prepare the experiment
import jax
import jax.numpy as jnp
import numpy as npThese imports provide JAX array transformations and NumPy reference calculations. Define the numerical functions next; the final build block supplies the concrete fixture and checks.
2 Implement the mechanism
Append this block in the same file. Follow the shape diagram above and identify each reduction or state transition.
# Step 2 — 2. Implement the mechanism
def layer_norm(x):
# Reduce along axis=-1 to compute `mean`.
mean = jnp.mean(x, axis=-1, keepdims=True)
# Reduce along axis=-1 to compute `variance`.
variance = jnp.mean((x-mean)**2, axis=-1, keepdims=True)
return (x-mean) / jnp.sqrt(variance + 1e-5)
def init_block(key, width=8):
# Create or split explicit PRNG key(s) (`keys`) for reproducible randomness.
keys = jax.random.split(key, 6)
shapes = [(width,width)]*4 + [(width,2*width),(2*width,width)]
return {name: jax.random.normal(k,s)*0.1 for name,k,s in zip(
["q","k","v","o","up","down"], keys, shapes)}
def block_forward(p, x):
h = layer_norm(x)
q, k, v = h@p["q"], h@p["k"], h@p["v"]
scores = q@k.T / jnp.sqrt(x.shape[-1])
allowed = jnp.arange(x.shape[0])[:,None] >= jnp.arange(x.shape[0])[None,:]
weights = jax.nn.softmax(jnp.where(allowed, scores, -jnp.inf), axis=-1)
residual = x + (weights@v)@p["o"]
return residual + jax.nn.gelu(layer_norm(residual)@p["up"])@p["down"]→ norm → Q/K/V → causal attention → output projection └────────────────────────────── add → residual residual → norm → expansion → GELU → contraction └────────────────────────────────── add → output
Explain the APIs in this step
jnp.mean(x, axis=..., keepdims=...): Averages JAX array values along the selected axis.jax.random.split(key, num=...): Derives new keys for independent random operations; using the same key again repeats its random stream.jax.random.normal(key, shape, dtype=...): Draws normal random values using an explicit PRNG key.jnp.arange(n, dtype=...): Creates a 1-D JAX array of evenly spaced values[0, 1, ..., n-1]on the target device.
3 Run and check
Append this block and run python main.py. Compare its output with your prediction. An assertion failure means a stated contract needs investigation.
# Step 3 — 3. Run and check
# Create or split explicit PRNG key(s) (`p`) for reproducible randomness.
p = init_block(jax.random.key(7))
x = jnp.arange(32, dtype=jnp.float32).reshape(4,8)/10
output = block_forward(p, x)
# Check tensor shape invariant: `output.shape == x.shape`
assert output.shape == x.shape
# Ensure all array elements remain finite: `jnp.all(jnp.isfinite(output))`
assert jnp.all(jnp.isfinite(output))
zero = jax.tree.map(jnp.zeros_like, p)
assert jnp.allclose(block_forward(zero, x), x)
assert jnp.allclose(jax.jit(block_forward)(p,x), output, atol=1e-5)
print("Block shape:", output.shape)Output shape ; finite values, zero-projection identity, compiled/eager agreement.
Explain the APIs in this step
jax.random.key(seed): Creates a typed explicit PRNG key from a seed.jnp.arange(n, dtype=...): Creates a 1-D JAX array of evenly spaced values[0, 1, ..., n-1]on the target device.array.reshape(new_shape): Reorganizes tensor axes without changing the total element count (array.size).jnp.isfinite(x): Returns a JAX boolean array marking elements that are neitherNaNnor infinite.
View the cumulative build in main.py
These blocks append in order to your separate practice file.
# Step 1 — 1. Prepare the experiment
import jax
import jax.numpy as jnp
import numpy as np
# Step 2 — 2. Implement the mechanism
def layer_norm(x):
# Reduce along axis=-1 to compute `mean`.
mean = jnp.mean(x, axis=-1, keepdims=True)
# Reduce along axis=-1 to compute `variance`.
variance = jnp.mean((x-mean)**2, axis=-1, keepdims=True)
return (x-mean) / jnp.sqrt(variance + 1e-5)
def init_block(key, width=8):
# Create or split explicit PRNG key(s) (`keys`) for reproducible randomness.
keys = jax.random.split(key, 6)
shapes = [(width,width)]*4 + [(width,2*width),(2*width,width)]
return {name: jax.random.normal(k,s)*0.1 for name,k,s in zip(
["q","k","v","o","up","down"], keys, shapes)}
def block_forward(p, x):
h = layer_norm(x)
q, k, v = h@p["q"], h@p["k"], h@p["v"]
scores = q@k.T / jnp.sqrt(x.shape[-1])
allowed = jnp.arange(x.shape[0])[:,None] >= jnp.arange(x.shape[0])[None,:]
weights = jax.nn.softmax(jnp.where(allowed, scores, -jnp.inf), axis=-1)
residual = x + (weights@v)@p["o"]
return residual + jax.nn.gelu(layer_norm(residual)@p["up"])@p["down"]
# Step 3 — 3. Run and check
# Create or split explicit PRNG key(s) (`p`) for reproducible randomness.
p = init_block(jax.random.key(7))
x = jnp.arange(32, dtype=jnp.float32).reshape(4,8)/10
output = block_forward(p, x)
# Check tensor shape invariant: `output.shape == x.shape`
assert output.shape == x.shape
# Ensure all array elements remain finite: `jnp.all(jnp.isfinite(output))`
assert jnp.all(jnp.isfinite(output))
zero = jax.tree.map(jnp.zeros_like, p)
assert jnp.allclose(block_forward(zero, x), x)
assert jnp.allclose(jax.jit(block_forward)(p,x), output, atol=1e-5)
print("Block shape:", output.shape)Run the example
python3 phases/07-transformers/03-a-transformer-block-and-mixed-precision/code/main.pyAbout this command
This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.
Inspect the complete assembled example
# Step 1 — 1. Prepare the experiment
import jax
import jax.numpy as jnp
import numpy as np
# Step 2 — 2. Implement the mechanism
def layer_norm(x):
# Normalize each token over its feature channels by computing its mean.
mean = jnp.mean(x, axis=-1, keepdims=True)
# Measure each token’s variance across its feature channels.
variance = jnp.mean((x-mean)**2, axis=-1, keepdims=True)
return (x-mean) / jnp.sqrt(variance + 1e-5)
def init_block(key, width=8):
# Create or split explicit PRNG key(s) (`keys`) for reproducible randomness.
keys = jax.random.split(key, 6)
shapes = [(width,width)]*4 + [(width,2*width),(2*width,width)]
return {name: jax.random.normal(k,s)*0.1 for name,k,s in zip(
["q","k","v","o","up","down"], keys, shapes)}
def block_forward(p, x):
h = layer_norm(x)
q, k, v = h@p["q"], h@p["k"], h@p["v"]
scores = q@k.T / jnp.sqrt(x.shape[-1])
allowed = jnp.arange(x.shape[0])[:,None] >= jnp.arange(x.shape[0])[None,:]
weights = jax.nn.softmax(jnp.where(allowed, scores, -jnp.inf), axis=-1)
residual = x + (weights@v)@p["o"]
return residual + jax.nn.gelu(layer_norm(residual)@p["up"])@p["down"]
# Step 3 — 3. Run and check
# Create or split explicit PRNG key(s) (`p`) for reproducible randomness.
p = init_block(jax.random.key(7))
x = jnp.arange(32, dtype=jnp.float32).reshape(4,8)/10
output = block_forward(p, x)
# Check tensor shape invariant: `output.shape == x.shape`
assert output.shape == x.shape
# Ensure all array elements remain finite: `jnp.all(jnp.isfinite(output))`
assert jnp.all(jnp.isfinite(output))
zero = jax.tree.map(jnp.zeros_like, p)
assert jnp.allclose(block_forward(zero, x), x)
assert jnp.allclose(jax.jit(block_forward)(p,x), output, atol=1e-5)
print("Block shape:", output.shape)Expected result: Output shape ; finite values, zero-projection identity, compiled/eager agreement.
Recorded output and runtime details
Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.
Block shape: (4, 8)
Block shape: (4, 8)
bfloat16 max absolute error: 0.013455629348754883
Float32 reduction policy max absolute error: 0.013455629348754883
PASS: transformers-03
Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.
Verify the residual identity and causal boundary
# Experiment — Verify the residual identity and causal boundary
changed_x = x.at[-1].add(100.)
changed_output = block_forward(p, changed_x)
assert jnp.allclose(changed_output[:3], output[:3], atol=1e-5)
assert not jnp.allclose(changed_output[-1], output[-1])Expected: First three rows agree; the last row changes.
Masking and per-token operations preserve causal independence through both sublayers.
Compare two precision policies
# Experiment — Compare two precision policies
low_p = jax.tree.map(lambda z:z.astype(jnp.bfloat16), p)
low_x = x.astype(jnp.bfloat16)
low_out = block_forward(low_p, low_x).astype(jnp.float32)
error = jnp.max(jnp.abs(low_out-output))
print("bfloat16 max absolute error:", float(error))
assert jnp.all(jnp.isfinite(low_out))
assert error < 0.1
def mixed_block(p, x):
h = layer_norm(x.astype(jnp.float32)).astype(jnp.bfloat16)
q,k,v = h@p["q"], h@p["k"], h@p["v"]
scores = (q@k.T).astype(jnp.float32)/jnp.sqrt(x.shape[-1])
allowed = jnp.arange(x.shape[0])[:,None] >= jnp.arange(x.shape[0])[None,:]
weights = jax.nn.softmax(jnp.where(allowed,scores,-jnp.inf),axis=-1).astype(jnp.bfloat16)
residual = x+(weights@v)@p["o"]
normalized = layer_norm(residual.astype(jnp.float32)).astype(jnp.bfloat16)
activation = jax.nn.gelu((normalized@p["up"]).astype(jnp.float32)).astype(jnp.bfloat16)
return residual + activation@p["down"]
mixed_out = mixed_block(low_p,low_x).astype(jnp.float32)
mixed_error = jnp.max(jnp.abs(mixed_out-output))
print("Float32 reduction policy max absolute error:",float(mixed_error))
# Ensure all array elements remain finite: `jnp.all(jnp.isfinite(mixed_out))`
assert jnp.all(jnp.isfinite(mixed_out))
assert mixed_error < 0.1Expected: Both outputs are finite, with max absolute errors below for this fixture. Record both actual errors; their relative order is not guaranteed.
Storage/activation casts and sensitive-reduction precision are separate choices. One fixture is distinct from an accuracy or throughput guarantee.
Make it yours
Where to put your code: Modify your own main.py, or use the Make it yours cell in your downloaded notebook. Run your attempt in the declared environment and verify your prediction before opening the reference solution.
Differentiate the mean squared block output with respect to every projection. Check each gradient leaf’s shape and finite values, then perturb one parameter coordinate and compare a central difference.
Show a writing guide and starter scaffold
How to write this exercise code
Learner-only inputs, outputs, checks and hints
Inputs
- The transformer block parameters, one sequence input, and a selected parameter coordinate with a finite-difference step.
Outputs
- Mean-squared output loss, gradient leaves with their shapes and finite checks, and the selected autodiff/central-difference comparison.
Invariants to check
- Every projection gradient matches its parameter shape and is finite; the central difference checks the selected loss derivative.
Hints
- Differentiate the scalar mean-squared block output with respect to the full parameter pytree.
- Perturb one scalar parameter coordinate in both directions and compare the resulting loss difference.
import jax
import jax.numpy as jnp
def check_block_parameter_gradients(params, inputs, parameter_path, coordinate, epsilon):
# TODO: Check every gradient leaf and compare one selected derivative with central differences.
raise NotImplementedErrorReveal the reference solution
# Exercise solution
def objective(params):
return jnp.mean(block_forward(params,x)**2)
g = jax.grad(objective)(p)
# Check tensor shape invariant: `all(a.shape==b.shape and jnp.all(jnp.isfinite(b)) for a,b in zip(...`
assert all(a.shape==b.shape and jnp.all(jnp.isfinite(b)) for a,b in zip(jax.tree.leaves(p),jax.tree.leaves(g)))
def shifted_objective(delta):
altered = {**p, "down": p["down"].at[0,0].add(delta)}
return objective(altered)
fd = (shifted_objective(1e-2)-shifted_objective(-1e-2))/(2e-2)
assert jnp.allclose(fd,g["down"][0,0],atol=1e-4,rtol=2e-2)Check normalization without autodiff
Transfer / diagnosis
Compute the first normalized row using NumPy and compare. Also check a constant row.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- Compute the first normalized row using NumPy and compare. Also check a constant row.
Outputs
- JAX/NumPy parity and finite output for a constant row.
Invariants to check
- Normalization uses the same epsilon and feature axis in both implementations.
Hints
- Use population variance over features and epsilon .
import jax
import jax.numpy as jnp
import numpy as np
def practice_check_normalization_without_autodiff(inputs):
"""Implement the check normalization without autodiff check."""
# TODO: Normalize the first row using a separate NumPy mean/variance calculation and check the constant-row case.
raise NotImplementedErrorShow a hint
Use population variance over features and epsilon .
Reveal reference solution and reasoning
# Check normalization without autodiff (Transfer / diagnosis)
row = np.asarray(x[0])
reference = (row-row.mean())/np.sqrt(np.mean((row-row.mean())**2)+1e-5)
np.testing.assert_allclose(layer_norm(x)[0],reference,atol=1e-6)
assert jnp.allclose(layer_norm(jnp.ones((2,8))),0.)The independent reduction and constant-row case check the normalization mechanism.
Repair a residual width mismatch
Transfer / diagnosis
Remove the contraction from the MLP and reproduce the residual addition failure. Restore the contraction and verify shape.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- Remove the contraction from the MLP and reproduce the residual addition failure. Restore the contraction and verify shape.
Outputs
- The incompatible-shape failure and repaired output shape.
Invariants to check
- Both residual operands share the same final feature dimension.
Hints
- The expanded tensor has twice the model width.
import numpy as np
def practice_repair_a_residual_width_mismatch(params, inputs):
"""Implement the repair a residual width mismatch check."""
# TODO: Remove the MLP contraction to reproduce the residual-add shape error, restore it, and assert both branches match residual width.
raise NotImplementedErrorShow a hint
The expanded tensor has twice the model width.
Reveal reference solution and reasoning
# Repair a residual width mismatch (Transfer / diagnosis)
expanded = jax.nn.gelu(layer_norm(x)@p["up"])
# Check tensor shape invariant: `expanded.shape == (4,16)`
assert expanded.shape == (4,16)
try:
_ = x + expanded
except TypeError:
pass
else:
raise AssertionError("expected width mismatch")
# Check tensor shape invariant: `(x + expanded@p["down"]).shape == x.shape`
assert (x + expanded@p["down"]).shape == x.shapeResidual paths require matching widths; the down projection restores the model-width contract.
Diagnose the result
A residual width error means the expanded MLP tensor was added before its contraction. Check versus , restore the down projection and verify the zero-projection identity. For nonfinite or inaccurate lower-precision results, inspect feature variance, epsilon and reduction dtype. Report both measured precision errors rather than assuming a policy must always improve them.
Carry forward
- Masking and per-token operations preserve causal independence through both sublayers.
- The error bound is a checked example, not an accuracy guarantee for arbitrary models.
Keep your evidence
Keep the architecture diagram, zero-projection identity, causal perturbation, independent normalization, finite-difference coordinate check, precision discrepancy and residual-width repair.
Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.
Saved in this browser. Export or back up your portfolio anytime in My learning.
Reference execution: CPU validation only. This worked reference is validated on CPU. Accelerator compatibility requires a separate source review and target execution receipt. Use the execution matrix to review device selection and data ownership before adapting a separate target program. Execution matrix and target qualification.