Phase 03 / State, randomness & control flow
Pytrees and structured parameters
Revision v0.18.1 · Updated since your last visit. Review the changes; your progress is saved.
Lesson · about 60 minutes · CPU practice
Represent parameters and state in a nested structure and transform its leaves.
Phase study guide: preparation, milestones and project evidence
Before you start
Before this: Random keys without surprises
Hardware: CPU
Setup help: Python, notebook, or Cloud TPU VM
Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions require separate checks. Hardware receipts apply only to their recorded source, environment, hardware, and cases; they do not qualify every lesson or TPU configuration. See the TPU performance and hardware evidence guide. The optional Cloud TPU specialization explains target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.
Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txtpy -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txtAfter activation, run the complete worked example from that same folder:
python phases/03-state/02-pytrees-and-structured-parameters/code/main.py.\.venv\Scripts\python.exe phases/03-state/02-pytrees-and-structured-parameters/code/main.pyFor an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.
Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.
For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:
python -m pip install jupyterlab
python -m jupyter lab.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter labSelect the downloaded state-02.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.
The worked reference for this lesson is CPU-validated. Some references explicitly configure logical CPU devices; setting JAX_PLATFORMS does not adapt them to a TPU. Follow the execution matrix and reviewed multi-host example before attempting a target run.
For accelerator preparation, study TPU setup, observed runtime receipts, checkpointed jobs and precision and retained traces. Current receipts cover bounded v5e1 launch, resume, and precision checks; four single-device Pallas cases; and a four-device v5e collective fixture. Each receipt is limited to its recorded source, environment, cases, and hardware. It does not qualify all lessons or TPU configurations. Cloud provisioning is optional for CPU learning.
Check: Before/after: , approximately .
Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.
What you will be able to do
- Identify containers, leaves and structure
- Derive the matching gradient tree
- Apply corresponding leaves, not unrelated arrays
- Keep metadata separate from differentiated state
The problem
Our model has a weight vector and a bias. How do we keep each parameter paired with its gradient? We’ll store them in a small nested structure called a pytree, calculate one update by hand, and check that JAX follows the same structure. The tree is just a way to keep related values organized.
The idea
A pytree is a nested structure whose leaves hold values such as parameters. JAX can transform the leaves while preserving the structure. This makes a model's parameter groups explicit and lets us compare the gradient tree with the parameters it differentiates.
- Parameter tree
{weight: array, bias: array}Names and containers describe the structure.
- Tree operation
map(update, parameters)Apply a matching operation to each array leaf.
- New parameter tree
same structure, new leavesThe optimizer and gradients must agree on the tree.
A pytree is a nested structure with array leaves, not one large flattened tensor. Shape and dtype still matter separately at each leaf.
Conceptual illustration. Verify the computation in the lesson’s Python experiments.
Match leaves by meaning as well as shape
A model can have several parameter groups, so the gradient must stay paired with the parameter it describes. A pytree preserves these relationships through dictionaries, lists and tuples while allowing transformations to act on array leaves. The container gives identity; each leaf holds a scalar or array.
In the diagram, follow each parameter path to its gradient partner. The two weight coordinates and bias are distinct values even if two leaves happen to share a shape. Optimizer state may add its own moment trees or counters, so do not assume every state object has exactly the parameter structure.
Identify containers, leaves and structure
Pytree structure matters because JAX transformations preserve the organization of nested inputs and outputs. In the example, the dictionary contains a weight vector of shape and scalar bias of shape : two leaves, three numerical parameter values. The leaf count is not the number of scalar parameters.
tree.flatten returns both an ordered list of leaves and a tree definition that can reconstruct their containers. Keep that definition with the values; manually guessing the flattening order risks reconnecting a number to the wrong parameter path.
params
├── weight: float32[2]
└── bias: float32[]
2 leaves, 3 numerical parameter valuesDerive the matching gradient tree
The gradient tree is easiest to trust when one update can be calculated by hand. With input , weights , bias , and target , prediction is , residual is , and squared loss is . The derivative with respect to prediction is ; multiplying by the input gives weight gradient , and the bias gradient is .
At step size , subtracting this gradient gives weights and bias . The updated prediction is and loss about , matching the execution receipt. Check each leaf’s value and role, not just the final loss decrease.
Apply corresponding leaves, not unrelated arrays
An elementwise update must pair each parameter leaf with its corresponding gradient leaf. tree.map can apply over two matching structures, but matching structure alone is not proof of semantic identity. A swapped pair of equal-shaped leaves may produce valid arithmetic and a wrong model.
Compare paths, shape and dtype before applying an update. A missing dictionary key is a tree-structure mismatch; incompatible leaf dimensions are a shape mismatch. Then verify an independently calculated prediction or update, as in the one-example check, to catch errors that structure checks cannot see.
Keep metadata separate from differentiated state
Differentiation needs numerical trainable leaves, while configuration and bookkeeping have different meanings. A model name string is not a floating-point parameter; keep it outside the differentiated tree unless a custom pytree explicitly defines how it participates.
Optimizer state is also not interchangeable with parameters. It may contain moment estimates shaped like the parameter leaves plus a scalar counter. Preserve those meanings and structures separately, then update each with its appropriate rule. The useful invariant is correspondence by path and role, not that every object in training has one identical tree.
A gradient tree matches parameter leaves
Predict before looking: Which weight changes more, and why?
Read the figure
The horizontal categories name weight coordinate one, weight coordinate two and bias; the vertical axis is parameter value. Each pair compares before and after one update: weights move from to , and bias from to . The second weight remains negative but moves upward toward zero.
Connect it to the computation
For input , the initial prediction is against target , giving gradient leaves . Subtracting that gradient with step produces the plotted increases . The updated prediction lowers loss from to about ; the chart shows parameter values, so verify loss separately.
Figure data and execution details
Open full-size figureDownload data and provenance
The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.
Your build: Create a separate main.py in your practice folder. Append each block in order and run python main.py in the lesson environment. The downloaded script is the complete reference, so keep it separate from your attempt.
1 Prepare the inputs
Create main.py in your lesson workspace. Add this first block; use the environment from setup.
# Step 1 — Prepare the inputs
import jax
import jax.numpy as jnpThese explicit inputs define the case that the later checks will verify.
2 Build the computation
Append this block below the inputs in the same file.
# Step 2 — Build the computation: loss reads both parameter leaves.
params = {"weight": jnp.array([1., -1.]), "bias": jnp.array(0.)}
x = jnp.array([2., 1.])
def loss(p):
prediction = jnp.dot(p["weight"], x) + p["bias"]
return (prediction - 3.) ** 2loss reads both parameter leaves. grad returns a matching gradient tree; tree.map pairs each parameter with its corresponding derivative for the update.
Explain the APIs in this step
jnp.array(values, dtype=...): Constructs a JAX array from Python/NumPy values.
3 Run and check the result
Append the checks, save main.py, and run python main.py from this folder using your course environment.
# Step 3 — Run and check the result
value, grads = jax.value_and_grad(loss)(params)
updated = jax.tree.map(lambda p, g: p - 0.05 * g, params, grads)
print("Before/after:", float(value), float(loss(updated)))
assert grads.keys() == params.keys()
assert loss(updated) < valueCompare the output to the expected result below before making the exercise change.
Explain the APIs in this step
jax.value_and_grad(loss_fn)(params, ...): Transforms a pure function to return its value and gradient(value, grads)from the same evaluation.jax.tree.map(fn, tree, ...): Applies a function leaf-by-leaf across matching PyTrees.
View the cumulative build in main.py
These blocks append in order to your separate practice file.
# Step 1 — Prepare the inputs
import jax
import jax.numpy as jnp
# Step 2 — Build the computation: loss reads both parameter leaves.
params = {"weight": jnp.array([1., -1.]), "bias": jnp.array(0.)}
x = jnp.array([2., 1.])
def loss(p):
prediction = jnp.dot(p["weight"], x) + p["bias"]
return (prediction - 3.) ** 2
# Step 3 — Run and check the result
value, grads = jax.value_and_grad(loss)(params)
updated = jax.tree.map(lambda p, g: p - 0.05 * g, params, grads)
print("Before/after:", float(value), float(loss(updated)))
assert grads.keys() == params.keys()
assert loss(updated) < valueRun the example
python3 phases/03-state/02-pytrees-and-structured-parameters/code/main.pyAbout this command
This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.
Inspect the complete assembled example
# Step 1 — Prepare the inputs
import jax
import jax.numpy as jnp
# Step 2 — Build the computation: loss reads both parameter leaves.
params = {"weight": jnp.array([1., -1.]), "bias": jnp.array(0.)}
x = jnp.array([2., 1.])
def loss(p):
prediction = jnp.dot(p["weight"], x) + p["bias"]
return (prediction - 3.) ** 2
# Step 3 — Run and check the result
value, grads = jax.value_and_grad(loss)(params)
updated = jax.tree.map(lambda p, g: p - 0.05 * g, params, grads)
print("Before/after:", float(value), float(loss(updated)))
assert grads.keys() == params.keys()
assert loss(updated) < valueExpected result: Before/after: , approximately .
Recorded output and runtime details
Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.
Before/after: 4.0 0.6399999260902405
Before/after: 4.0 0.6399999260902405
Expected tree mismatch
PASS: state-02
Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.
Verify every gradient leaf
# Experiment — Verify every gradient leaf
assert jnp.allclose(grads["weight"],jnp.array([-8.,-4.]))
assert jnp.allclose(grads["bias"],-4.)
assert jnp.allclose(updated["weight"],jnp.array([1.4,-0.8]))
assert jnp.allclose(updated["bias"],0.2)
assert jnp.allclose(loss(updated),0.64,atol=1e-6)Expected: All leaves agree with the hand-derived update.
This check verifies parameter identity and the actual gradient values, not just a matching container.
Flatten and reconstruct without losing identity
# Experiment — Flatten and reconstruct without losing identity
leaves,definition=jax.tree.flatten(params)
rebuilt=jax.tree.unflatten(definition,leaves)
assert jax.tree.structure(rebuilt)==jax.tree.structure(params)
assert jnp.array_equal(rebuilt["weight"],params["weight"])
assert jnp.array_equal(rebuilt["bias"],params["bias"])Expected: The original keys and corresponding values are reconstructed.
The tree definition describes containers; leaves supply numerical values. Both are needed for reconstruction.
Make it yours
Where to put your code: Modify your own main.py, or use the Make it yours cell in your downloaded notebook. Run your attempt in the declared environment and verify your prediction before opening the reference solution.
Compute the squared norm of all gradient leaves. Verify it by adding the weight and bias contributions directly.
Show a writing guide and starter scaffold
How to write this exercise code
Learner-only inputs, outputs, checks and hints
Inputs
- A nested parameter pytree with weight and bias leaves and a matching gradient pytree.
Outputs
- The squared Euclidean norm of all gradient leaves.
Invariants to check
- Every gradient leaf contributes exactly once.
- The pytree structure is preserved while reducing the scalar contributions.
Hints
- Map a square operation over gradient leaves, then reduce the resulting leaves.
- Compare your result to direct weight and bias contributions.
import jax
import jax.numpy as jnp
def squared_gradient_norm(grads: dict) -> jax.Array:
"""Return the sum of squared values across all gradient leaves."""
# TODO: traverse the pytree leaves and reduce their squared entries.
raise NotImplementedErrorReveal the reference solution
# Exercise solution: Compute the squared norm of all gradient leaves.
norm_squared = sum(jnp.sum(g ** 2) for g in jax.tree.leaves(grads))
manual = jnp.sum(grads["weight"] ** 2) + grads["bias"] ** 2
assert jnp.allclose(norm_squared, manual)
assert jnp.allclose(norm_squared, 96.)Extend to a nested model
Practice
Place weight and bias under a layer dictionary and add a scalar scale parameter. Verify the derivative of scale for .
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- A weight vector, scalar bias, scalar scale, and input vector.
Outputs
- The gradient pytree for the squared prediction loss of scale*(weight dot x + bias).
Invariants to check
- Gradient leaves follow the same nested structure as the parameters.
- The scale derivative is computed by autodiff for the supplied values.
Hints
- Put weight and bias below a layer key, with scale as a separate leaf.
- Differentiate a scalar loss with jax.grad.
import jax
import jax.numpy as jnp
def nested_gradients(weight: jax.Array, bias: jax.Array, scale: jax.Array, x: jax.Array, target: jax.Array) -> dict:
"""Return gradients for a nested scaled linear model."""
# TODO: construct the parameter pytree and differentiate its scalar loss.
raise NotImplementedErrorShow a hint
At the starting point prediction is and residual is , so the scale derivative is .
Reveal reference solution and reasoning
# Extend to a nested model (Practice): Nested parameter identity survives differentiation.
nested={"layer":params,"scale":jnp.array(1.)}
def nested_loss(p):
pred=p["scale"]*(jnp.dot(p["layer"]["weight"],x)+p["layer"]["bias"])
return (pred-3.)**2
g=jax.grad(nested_loss)(nested)
assert jax.tree.structure(g)==jax.tree.structure(nested)
assert jnp.allclose(g["scale"],-4.)Nested parameter identity survives differentiation. The independent scale calculation checks the new path.
Repair a mismatched update tree
Challenge
Remove bias from the gradient dictionary, reproduce the structural error, then restore it and verify the hand update.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- A parameter pytree and a matching gradient pytree.
Outputs
- A gradient tree validated against the parameter structure and a matching updated parameter pytree.
Invariants to check
- Every parameter path has exactly one gradient leaf.
- A missing bias gradient is rejected instead of broadcast or silently omitted.
Hints
- Use tree structure comparison or a tree map to validate alignment.
- Apply the hand update only after the structures match.
import jax
import jax.numpy as jnp
def checked_tree_update(params: dict, grads: dict, learning_rate: float = 0.05) -> dict:
"""Reject a mismatched pytree and otherwise perform one aligned update."""
# TODO: check structure compatibility, then update corresponding leaves.
raise NotImplementedErrorShow a hint
Matching leaf counts is not enough; key paths matter.
Reveal reference solution and reasoning
# Repair a mismatched update tree (Challenge): A missing bias gradient is a tree mismatch.
try:
jax.tree.map(lambda p,g:p-0.05*g,params,{"weight":grads["weight"]})
except ValueError:
print("Expected tree mismatch")
else:
raise AssertionError("Expected a missing-key failure")
fixed=jax.tree.map(lambda p,g:p-0.05*g,params,grads)
assert jnp.allclose(loss(fixed),0.64,atol=1e-6)A missing bias gradient is a tree mismatch. Broadcasting weights or changing a learning rate cannot supply the missing parameter identity.
Diagnose the result
A missing bias gradient is a tree mismatch. Broadcasting weights or changing a learning rate cannot supply the missing parameter identity.
Carry forward
- This check verifies parameter identity and the actual gradient values, not just a matching container.
- The tree definition describes containers; leaves supply numerical values. Both are needed for reconstruction.
Keep your evidence
Keep the parameter/gradient tree diagram, hand-derived leaves and update, flatten/unflatten round trip, nested-scale derivative and missing-key repair.
Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.
Saved in this browser. Export or back up your portfolio anytime in My learning.
Reference execution: CPU validation only. This worked reference is validated on CPU. Accelerator compatibility requires a separate source review and target execution receipt. Use the execution matrix to review device selection and data ownership before adapting a separate target program. Execution matrix and target qualification.