Core Foundation · Local CPULaptop CPULesson 2 of 4 in Phase 03

Phase 03 / State, randomness & control flow

Pytrees and structured parameters

Revision v0.18.1 ·

Lesson · about 60 minutes · CPU practice

Represent parameters and state in a nested structure and transform its leaves.

Phase study guide: preparation, milestones and project evidence

Before you start

Before this: Random keys without surprises
Hardware: CPU

Setup help: Python, notebook, or Cloud TPU VM

Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions require separate checks. Hardware receipts apply only to their recorded source, environment, hardware, and cases; they do not qualify every lesson or TPU configuration. See the TPU performance and hardware evidence guide. The optional Cloud TPU specialization explains target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.

Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.

macOS / Linux terminal · repository folder
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txt

PowerShell · repository folder
py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txt

After activation, run the complete worked example from that same folder:

macOS / Linux terminal · activated environment
python phases/03-state/02-pytrees-and-structured-parameters/code/main.py

PowerShell command · repository folder
.\.venv\Scripts\python.exe phases/03-state/02-pytrees-and-structured-parameters/code/main.py

For an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.

Check: Before/after: 4.04.0, approximately 0.640.64.

Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.

Stage 1 of 5 · Understand the MechanismCore mental model, equations & visual walkthrough

What you will be able to do

  • Identify containers, leaves and structure
  • Derive the matching gradient tree
  • Apply corresponding leaves, not unrelated arrays
  • Keep metadata separate from differentiated state

The problem

Our model has a weight vector and a bias. How do we keep each parameter paired with its gradient? We’ll store them in a small nested structure called a pytree, calculate one update by hand, and check that JAX follows the same structure. The tree is just a way to keep related values organized.

The idea

A pytree is a nested structure whose leaves hold values such as parameters. JAX can transform the leaves while preserving the structure. This makes a model's parameter groups explicit and lets us compare the gradient tree with the parameters it differentiates.

Structure travels with the leaves
  1. Parameter tree{weight: array, bias: array}

    Names and containers describe the structure.

  2. Tree operationmap(update, parameters)

    Apply a matching operation to each array leaf.

  3. New parameter treesame structure, new leaves

    The optimizer and gradients must agree on the tree.

A pytree is a nested structure with array leaves, not one large flattened tensor. Shape and dtype still matter separately at each leaf.

Conceptual illustration. Verify the computation in the lesson’s Python experiments.

Match leaves by meaning as well as shape

A model can have several parameter groups, so the gradient must stay paired with the parameter it describes. A pytree preserves these relationships through dictionaries, lists and tuples while allowing transformations to act on array leaves. The container gives identity; each leaf holds a scalar or array.

In the diagram, follow each parameter path to its gradient partner. The two weight coordinates and bias are distinct values even if two leaves happen to share a shape. Optimizer state may add its own moment trees or counters, so do not assume every state object has exactly the parameter structure.

Identify containers, leaves and structure

Pytree structure matters because JAX transformations preserve the organization of nested inputs and outputs. In the example, the dictionary contains a weight vector of shape (2,)(2,) and scalar bias of shape ()(): two leaves, three numerical parameter values. The leaf count is not the number of scalar parameters.

tree.flatten returns both an ordered list of leaves and a tree definition that can reconstruct their containers. Keep that definition with the values; manually guessing the flattening order risks reconnecting a number to the wrong parameter path.

params
├── weight: float32[2]
└── bias: float32[]
2 leaves, 3 numerical parameter values

Derive the matching gradient tree

The gradient tree is easiest to trust when one update can be calculated by hand. With input (2,1)(2,1), weights (1,−1)(1,-1), bias 00, and target 33, prediction is 11, residual is −2-2, and squared loss is 44. The derivative with respect to prediction is −4-4; multiplying by the input gives weight gradient (−8,−4)(-8,-4), and the bias gradient is −4-4.

At step size 0.050.05, subtracting this gradient gives weights (1.4,−0.8)(1.4,-0.8) and bias 0.20.2. The updated prediction is 2.22.2 and loss about 0.640.64, matching the execution receipt. Check each leaf’s value and role, not just the final loss decrease.

y^=2(1)+1(−1)+0=1L=(1−3)2=4∇wL=(−8,−4),∂L∂b=−4\begin{aligned}\hat y&=2(1)+1(-1)+0=1\\L&=(1-3)^2=4\\\nabla_w L&=(-8,-4),\qquad \frac{\partial L}{\partial b}=-4\end{aligned}

Apply corresponding leaves, not unrelated arrays

An elementwise update must pair each parameter leaf with its corresponding gradient leaf. tree.map can apply p−ηgp-\eta g over two matching structures, but matching structure alone is not proof of semantic identity. A swapped pair of equal-shaped leaves may produce valid arithmetic and a wrong model.

Compare paths, shape and dtype before applying an update. A missing dictionary key is a tree-structure mismatch; incompatible leaf dimensions are a shape mismatch. Then verify an independently calculated prediction or update, as in the one-example check, to catch errors that structure checks cannot see.

Keep metadata separate from differentiated state

Differentiation needs numerical trainable leaves, while configuration and bookkeeping have different meanings. A model name string is not a floating-point parameter; keep it outside the differentiated tree unless a custom pytree explicitly defines how it participates.

Optimizer state is also not interchangeable with parameters. It may contain moment estimates shaped like the parameter leaves plus a scalar counter. Preserve those meanings and structures separately, then update each with its appropriate rule. The useful invariant is correspondence by path and role, not that every object in training has one identical tree.

Read the figure

The horizontal categories name weight coordinate one, weight coordinate two and bias; the vertical axis is parameter value. Each pair compares before and after one update: weights move from (1,−1)(1,-1) to (1.4,−0.8)(1.4,-0.8), and bias from 00 to 0.20.2. The second weight remains negative but moves upward toward zero.

Connect it to the computation

For input (2,1)(2,1), the initial prediction is 11 against target 33, giving gradient leaves (−8,−4,−4)(-8,-4,-4). Subtracting that gradient with step 0.050.05 produces the plotted increases (0.4,0.2,0.2)(0.4,0.2,0.2). The updated prediction 2.22.2 lowers loss from 44 to about 0.640.64; the chart shows parameter values, so verify loss separately.

Figure data and execution details

Recorded environment: JAX 0.9.2 · 1 CPU device

The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.

Stage 2 of 5 · Build & Run the ReferenceStep-by-step code construction & verified execution output

Your build: Create a separate main.py in your practice folder. Append each block in order and run python main.py in the lesson environment. The downloaded script is the complete reference, so keep it separate from your attempt.

1 Prepare the inputs

Create main.py in your lesson workspace. Add this first block; use the environment from setup.

main.py · append build step 1
# Step 1 — Prepare the inputs
import jax
import jax.numpy as jnp

These explicit inputs define the case that the later checks will verify.

2 Build the computation

Append this block below the inputs in the same file.

main.py · append build step 2
# Step 2 — Build the computation: loss reads both parameter leaves.
params = {"weight": jnp.array([1., -1.]), "bias": jnp.array(0.)}
x = jnp.array([2., 1.])
def loss(p):
    prediction = jnp.dot(p["weight"], x) + p["bias"]
    return (prediction - 3.) ** 2

loss reads both parameter leaves. grad returns a matching gradient tree; tree.map pairs each parameter with its corresponding derivative for the update.

Explain the APIs in this step
  • jnp.array(values, dtype=...): Constructs a JAX array from Python/NumPy values.

3 Run and check the result

Append the checks, save main.py, and run python main.py from this folder using your course environment.

main.py · append build step 3
# Step 3 — Run and check the result
value, grads = jax.value_and_grad(loss)(params)
updated = jax.tree.map(lambda p, g: p - 0.05 * g, params, grads)
print("Before/after:", float(value), float(loss(updated)))
assert grads.keys() == params.keys()
assert loss(updated) < value

Compare the output to the expected result below before making the exercise change.

Explain the APIs in this step
  • jax.value_and_grad(loss_fn)(params, ...): Transforms a pure function to return its value and gradient (value, grads) from the same evaluation.
  • jax.tree.map(fn, tree, ...): Applies a function leaf-by-leaf across matching PyTrees.
View the cumulative build in main.py

These blocks append in order to your separate practice file.

main.py · cumulative Python build
# Step 1 — Prepare the inputs
import jax
import jax.numpy as jnp

# Step 2 — Build the computation: loss reads both parameter leaves.
params = {"weight": jnp.array([1., -1.]), "bias": jnp.array(0.)}
x = jnp.array([2., 1.])
def loss(p):
    prediction = jnp.dot(p["weight"], x) + p["bias"]
    return (prediction - 3.) ** 2

# Step 3 — Run and check the result
value, grads = jax.value_and_grad(loss)(params)
updated = jax.tree.map(lambda p, g: p - 0.05 * g, params, grads)
print("Before/after:", float(value), float(loss(updated)))
assert grads.keys() == params.keys()
assert loss(updated) < value

Run the example

python3 phases/03-state/02-pytrees-and-structured-parameters/code/main.py
About this command

This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.

Inspect the complete assembled example
main.py · complete Python example
# Step 1 — Prepare the inputs
import jax
import jax.numpy as jnp
# Step 2 — Build the computation: loss reads both parameter leaves.
params = {"weight": jnp.array([1., -1.]), "bias": jnp.array(0.)}
x = jnp.array([2., 1.])
def loss(p):
    prediction = jnp.dot(p["weight"], x) + p["bias"]
    return (prediction - 3.) ** 2
# Step 3 — Run and check the result
value, grads = jax.value_and_grad(loss)(params)
updated = jax.tree.map(lambda p, g: p - 0.05 * g, params, grads)
print("Before/after:", float(value), float(loss(updated)))
assert grads.keys() == params.keys()
assert loss(updated) < value

Expected result: Before/after: 4.04.0, approximately 0.640.64.

Recorded output and runtime details

Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.

2026-10-27T17:03:07.732705+00:00 · Python 3.14.3 · JAX 0.9.2 · 1 CPU device

Before/after: 4.0 0.6399999260902405
Before/after: 4.0 0.6399999260902405
Expected tree mismatch
PASS: state-02

Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.

Stage 3 of 5 · Controlled ExperimentsPredict before running & test boundary conditions

Verify every gradient leaf

Verify every gradient leaf · Python experiment
# Experiment — Verify every gradient leaf
assert jnp.allclose(grads["weight"],jnp.array([-8.,-4.]))
assert jnp.allclose(grads["bias"],-4.)
assert jnp.allclose(updated["weight"],jnp.array([1.4,-0.8]))
assert jnp.allclose(updated["bias"],0.2)
assert jnp.allclose(loss(updated),0.64,atol=1e-6)

Expected: All leaves agree with the hand-derived update.

This check verifies parameter identity and the actual gradient values, not just a matching container.

Flatten and reconstruct without losing identity

Flatten and reconstruct without losing identity · Python experiment
# Experiment — Flatten and reconstruct without losing identity
leaves,definition=jax.tree.flatten(params)
rebuilt=jax.tree.unflatten(definition,leaves)
assert jax.tree.structure(rebuilt)==jax.tree.structure(params)
assert jnp.array_equal(rebuilt["weight"],params["weight"])
assert jnp.array_equal(rebuilt["bias"],params["bias"])

Expected: The original keys and corresponding values are reconstructed.

The tree definition describes containers; leaves supply numerical values. Both are needed for reconstruction.

Stage 4 of 5 · Hands-On Exercise & CheckpointModify the code yourself, solve transfer tasks & test your understanding
Hands-On Exercise · Modify & Verify Run in the declared lesson environment

Make it yours

Where to put your code: Modify your own main.py, or use the Make it yours cell in your downloaded notebook. Run your attempt in the declared environment and verify your prediction before opening the reference solution.

Compute the squared norm of all gradient leaves. Verify it by adding the weight and bias contributions directly.

Show a writing guide and starter scaffold

How to write this exercise code

Learner-only inputs, outputs, checks and hints

Inputs

  • A nested parameter pytree with weight and bias leaves and a matching gradient pytree.

Outputs

  • The squared Euclidean norm of all gradient leaves.

Invariants to check

  • Every gradient leaf contributes exactly once.
  • The pytree structure is preserved while reducing the scalar contributions.

Hints

  • Map a square operation over gradient leaves, then reduce the resulting leaves.
  • Compare your result to direct weight and bias contributions.
Your attempt · complete the TODOs
import jax
import jax.numpy as jnp

def squared_gradient_norm(grads: dict) -> jax.Array:
    """Return the sum of squared values across all gradient leaves."""
    # TODO: traverse the pytree leaves and reduce their squared entries.
    raise NotImplementedError
Reveal the reference solution
Reference solution
# Exercise solution: Compute the squared norm of all gradient leaves.
norm_squared = sum(jnp.sum(g ** 2) for g in jax.tree.leaves(grads))
manual = jnp.sum(grads["weight"] ** 2) + grads["bias"] ** 2
assert jnp.allclose(norm_squared, manual)
assert jnp.allclose(norm_squared, 96.)

Extend to a nested model

Practice

Place weight and bias under a layer dictionary and add a scalar scale parameter. Verify the derivative of scale for s(wx+b)s(wx+b).

Show a writing guide and starter scaffold

How to write this practice solution

Learner-only inputs, outputs, checks and hints

Inputs

  • A weight vector, scalar bias, scalar scale, and input vector.

Outputs

  • The gradient pytree for the squared prediction loss of scale*(weight dot x + bias).

Invariants to check

  • Gradient leaves follow the same nested structure as the parameters.
  • The scale derivative is computed by autodiff for the supplied values.

Hints

  • Put weight and bias below a layer key, with scale as a separate leaf.
  • Differentiate a scalar loss with jax.grad.
Your attempt · complete the TODOs
import jax
import jax.numpy as jnp

def nested_gradients(weight: jax.Array, bias: jax.Array, scale: jax.Array, x: jax.Array, target: jax.Array) -> dict:
    """Return gradients for a nested scaled linear model."""
    # TODO: construct the parameter pytree and differentiate its scalar loss.
    raise NotImplementedError
Show a hint

At the starting point prediction is 11 and residual is −2-2, so the scale derivative is −4-4.

Reveal reference solution and reasoning
Extend to a nested model · Python solution
# Extend to a nested model (Practice): Nested parameter identity survives differentiation.
nested={"layer":params,"scale":jnp.array(1.)}
def nested_loss(p):
    pred=p["scale"]*(jnp.dot(p["layer"]["weight"],x)+p["layer"]["bias"])
    return (pred-3.)**2
g=jax.grad(nested_loss)(nested)
assert jax.tree.structure(g)==jax.tree.structure(nested)
assert jnp.allclose(g["scale"],-4.)

Nested parameter identity survives differentiation. The independent scale calculation checks the new path.

Repair a mismatched update tree

Challenge

Remove bias from the gradient dictionary, reproduce the structural error, then restore it and verify the hand update.

Show a writing guide and starter scaffold

How to write this practice solution

Learner-only inputs, outputs, checks and hints

Inputs

  • A parameter pytree and a matching gradient pytree.

Outputs

  • A gradient tree validated against the parameter structure and a matching updated parameter pytree.

Invariants to check

  • Every parameter path has exactly one gradient leaf.
  • A missing bias gradient is rejected instead of broadcast or silently omitted.

Hints

  • Use tree structure comparison or a tree map to validate alignment.
  • Apply the hand update only after the structures match.
Your attempt · complete the TODOs
import jax
import jax.numpy as jnp

def checked_tree_update(params: dict, grads: dict, learning_rate: float = 0.05) -> dict:
    """Reject a mismatched pytree and otherwise perform one aligned update."""
    # TODO: check structure compatibility, then update corresponding leaves.
    raise NotImplementedError
Show a hint

Matching leaf counts is not enough; key paths matter.

Reveal reference solution and reasoning
Repair a mismatched update tree · Python solution
# Repair a mismatched update tree (Challenge): A missing bias gradient is a tree mismatch.
try:
    jax.tree.map(lambda p,g:p-0.05*g,params,{"weight":grads["weight"]})
except ValueError:
    print("Expected tree mismatch")
else:
    raise AssertionError("Expected a missing-key failure")
fixed=jax.tree.map(lambda p,g:p-0.05*g,params,grads)
assert jnp.allclose(loss(fixed),0.64,atol=1e-6)

A missing bias gradient is a tree mismatch. Broadcasting weights or changing a learning rate cannot supply the missing parameter identity.

Check your understanding

What structure does grad(loss)(params) have?

Choose one answer

Diagnose the result

A missing bias gradient is a tree mismatch. Broadcasting weights or changing a learning rate cannot supply the missing parameter identity.

Stage 5 of 5 · Keep Your EvidenceRecord your verification receipt & carry takeaways forward

Carry forward

  • This check verifies parameter identity and the actual gradient values, not just a matching container.
  • The tree definition describes containers; leaves supply numerical values. Both are needed for reconstruction.

Keep your evidence

Download evidence template

Keep the parameter/gradient tree diagram, hand-derived leaves and update, flatten/unflatten round trip, nested-scale derivative and missing-key repair.

Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.

Saved in this browser. Export or back up your portfolio anytime in My learning.

Save work for this lesson

Reference execution: CPU validation only. This worked reference is validated on CPU. Accelerator compatibility requires a separate source review and target execution receipt. Use the execution matrix to review device selection and data ownership before adapting a separate target program. Execution matrix and target qualification.

Primary references