Phase 14 / Autodiff & JAX internals
Custom derivative rules
Revision v0.18.1 · Updated since your last visit. Review the changes; your progress is saved.
Lesson · about 120 minutes · CPU practice
Implement actual custom_jvp and custom_vjp rules with correct return and residual contracts. Verify seed linearity, stable extreme values and second derivatives.
Phase study guide: preparation, milestones and project evidence
Before you start
Before this: JVPs, VJPs, and higher-order derivatives
Hardware: CPU
Setup help: Python, notebook, or Cloud TPU VM
Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions require separate checks. Hardware receipts apply only to their recorded source, environment, hardware, and cases; they do not qualify every lesson or TPU configuration. See the TPU performance and hardware evidence guide. The optional Cloud TPU specialization explains target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.
Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txtpy -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txtAfter activation, run the complete worked example from that same folder:
python phases/14-internals/03-custom-derivative-rules/code/main.py.\.venv\Scripts\python.exe phases/14-internals/03-custom-derivative-rules/code/main.pyFor an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.
Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.
For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:
python -m pip install jupyterlab
python -m jupyter lab.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter labSelect the downloaded internals-03.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.
The worked reference for this lesson is CPU-validated. Some references explicitly configure logical CPU devices; setting JAX_PLATFORMS does not adapt them to a TPU. Follow the execution matrix and reviewed multi-host example before attempting a target run.
For accelerator preparation, study TPU setup, observed runtime receipts, checkpointed jobs and precision and retained traces. Current receipts cover bounded v5e1 launch, resume, and precision checks; four single-device Pallas cases; and a four-device v5e collective fixture. Each receipt is limited to its recorded source, environment, cases, and hardware. It does not qualify all lessons or TPU configurations. Cloud provisioning is optional for CPU learning.
Check: Softplus derivatives at zero are 0.5 and 0.25. Stable values at [-1000, 0, 1000] are [0, log(2), 1000]; the naive positive extreme overflows. The custom reverse Hessian matches diag(p)-p p-transpose.
Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.
What you will be able to do
- Implement actual custom_jvp and custom_vjp rules with correct return and residual contracts.
- Verify seed linearity, stable extreme values and second derivatives.
- Detect a wrong but linear derivative rule using independent numerical checks.
- Explain the forward-mode limitation of a custom_vjp function.
The problem
Sometimes you know a stable formula for a derivative, or need to control what a reverse pass saves. How do you replace JAX’s default differentiation without quietly changing the mathematics? We will write two exact custom rules, check higher derivatives, and catch a deliberately plausible wrong rule.
The idea
A custom derivative rule changes how differentiation treats a function. It must preserve the intended derivative while respecting transformation requirements. Stability of the primal computation and correctness of the derivative need separate checks.
A custom rule is a mathematical contract
Start from a known analytic function and derive its slope. Compare the custom rule with that result over ordinary and extreme inputs, then examine the second derivative if higher-order use is intended.
A forward rule must be linear in its tangent input; a reverse rule must be linear in its incoming cotangent. Inserting an arbitrary nonlinear operation on that sensitivity can produce a rule that no longer represents a derivative.
The slope and curvature curves show more than a successful first gradient at one point. Follow their behavior near numerically difficult regions and use finite differences only where that independent approximation is reliable.
Keep the primal stable before customizing its derivative
Softplus is . Direct exponentiation overflows for a large positive input, even though the correct output is approximately . logaddexp computes the same mathematical function without forming the dangerous exponential. Its derivative is the sigmoid , and its second derivative is . We use these identities as checks, not as permission to ignore the primal calculation.
For the executed points , the custom rule’s slopes are approximately , while the curvatures are . The peak curvature at zero and smaller tail values give a concrete second-derivative check; comparing only one slope could miss an incorrect curvature rule. The separate extreme-value probe returns softplus(1000)=1000 while direct log(1+exp(x)) overflows to infinity.
A custom JVP must remain linear in its tangent
The JVP receives tuples of primal arguments and tangent arguments. For one scalar input, it returns . The coefficient depends on the primal point; the expression is linear in the tangent . Squaring the tangent would not represent a derivative and would prevent a valid linear transpose. Multiplying by an incorrect constant remains linear but is still mathematically wrong, so linearity checks alone is distinct from correctness.
Choose residuals for a custom reverse pass
For , the gradient is softmax . The forward rule returns the scalar primal and saves as the residual. The backward rule receives that residual and a scalar output cotangent , then returns the input cotangent inside a one-element tuple. Saving probabilities is a deliberate memory/recomputation choice in this small example; it is not a universal optimal residual policy.
Higher-order behavior is part of the promise
Differentiating the softplus rule again should give curvature at zero. Differentiating the log-sum-exp reverse gradient gives , a symmetric Hessian whose rows sum to zero because adding a common constant changes the value linearly. The reference checks use these formulas. A stop_gradient inserted into a residual path might preserve first derivatives while breaking these higher-order checks.
Custom reverse mode has a transformation boundary
Applying jvp directly to a custom_vjp function is unsupported in the tested JAX API. We reproduce that boundary and report it explicitly. Reverse-over-reverse differentiation of this smooth, correctly implemented rule works and is checked. If forward-mode composition is needed, use an appropriate custom_jvp or ordinary JAX implementation rather than assuming every derivative decorator is interchangeable.
An intentionally different gradient needs a different claim
A straight-through or surrogate rule can be useful in a modeling method, but it is not the exact derivative of the reported primal. This lesson teaches exact custom derivatives. We deliberately halve a correct derivative to show that a smooth-looking result and seed linearity do not make it true. A finite-difference comparison away from flat or saturated regions exposes the error.
A stable custom rule preserves both slope and curvature
Predict before looking: Where should softplus have the greatest curvature, and what happens far into either tail?
Read the figure
The horizontal axis is input from to ; the upper vertical axis is slope and the lower is curvature. The custom-rule curves rise from near zero to slope 0.5 and curvature 0.25 at ; curvature is symmetric and falls to about 0.01766 at . That peak is the best place in this fixture to catch a wrong curvature coefficient. In the tails, predict a slope near 0 on the left and near 1 on the right, with curvature near 0 in both.
Connect it to the computation
The plotted derivatives come from differentiating the registered custom JVP rule once and twice; the execution output independently reports slopes and curvatures at . The extreme test is outside the plot: stable softplus returns 1000 at input 1000, while the naive expression returns infinity. Treat these as separate checks of smooth derivative behavior and primal stability.
Figure data and execution details
Open full-size figureDownload data and provenance
The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.
Your build: Create a separate main.py in your practice folder. Append each block in order and run python main.py in the lesson environment. The downloaded script is the complete reference, so keep it separate from your attempt.
1 Write stable primal functions and exact derivative contracts
Create main.py for the first block, then append subsequent blocks in order using your course CPU environment.
# Step 1 — Write stable primal functions and exact derivative contracts
import jax
# Enable 64-bit arithmetic before importing jax.numpy or creating arrays.
jax.config.update("jax_enable_x64", True)
import jax.numpy as jnp
import numpy as np
# Register custom derivative rule on `stable_softplus(x)`:
@jax.custom_jvp
def stable_softplus(x):
return jnp.logaddexp(0.0,x)
@stable_softplus.defjvp
def softplus_jvp(primals,tangents):
x,=primals
dx,=tangents
return stable_softplus(x),jax.nn.sigmoid(x)*dx
# Register custom derivative rule on `stable_logsumexp(x)`:
@jax.custom_vjp
def stable_logsumexp(x):
return jax.scipy.special.logsumexp(x)
def logsumexp_fwd(x):
value=stable_logsumexp(x)
weights=jax.nn.softmax(x)
return value,weights
def logsumexp_bwd(weights,cotangent):
return (cotangent*weights,)
stable_logsumexp.defvjp(logsumexp_fwd,logsumexp_bwd)The JVP rule returns primal and tangent; the VJP forward rule returns primal and residual, and its backward rule returns one cotangent per original input. Both preserve linearity in the directional seed.
2 Check first and second derivatives independently
Create main.py for the first block, then append subsequent blocks in order using your course CPU environment.
# Step 2 — Check first and second derivatives independently
points=jnp.array([-4.,0.,3.])
# Vectorize across the batch dimension with `jax.vmap` (`values`).
values=jax.vmap(stable_softplus)(points)
first=jax.vmap(jax.grad(stable_softplus))(points)
second=jax.vmap(jax.grad(jax.grad(stable_softplus)))(points)
# Transfer the device result to the host for an independent NumPy comparison.
reference_first=1/(1+np.exp(-np.asarray(points)))
reference_second=reference_first*(1-reference_first)
# Transfer the device result to the host for an independent NumPy comparison.
np.testing.assert_allclose(values,np.logaddexp(0,np.asarray(points)),rtol=1e-12)
np.testing.assert_allclose(first,reference_first,rtol=1e-12)
np.testing.assert_allclose(second,reference_second,rtol=1e-12)
x=jnp.array([-0.8,0.4,1.1])
# Transfer the device result to the host for an independent NumPy comparison.
# Accumulate the next contribution into `weights`.
weights=np.exp(np.asarray(x)-np.max(np.asarray(x)))
weights/=weights.sum()
hessian_reference=np.diag(weights)-np.outer(weights,weights)
np.testing.assert_allclose(jax.grad(stable_logsumexp)(x),weights,rtol=1e-12)
np.testing.assert_allclose(jax.jacrev(jax.grad(stable_logsumexp))(x),hessian_reference,rtol=1e-12,atol=1e-12)
print("Softplus first/second:",np.asarray(first),np.asarray(second))
print("Logsumexp gradient:",weights)An exact first derivative can still hide mistakes in a higher-order rule. The softplus curvature and logsumexp Hessian are independently derived here.
Explain the APIs in this step
jnp.array(values, dtype=...): Constructs a JAX array from Python/NumPy values.jax.vmap(fn, in_axes=..., out_axes=...): Vectorizes a single-example function across mapped array axes without a Python loop.jax.grad(loss_fn)(params, ...): Transforms a scalar-output function into one returning the gradient PyTree with the same structure asparams.np.testing.assert_allclose(actual, desired, rtol=..., atol=...): Checks a host-side numerical expectation and raisesAssertionErrorwhen values differ beyond tolerance.
3 Exercise extremes and seed linearity
Create main.py for the first block, then append subsequent blocks in order using your course CPU environment.
# Step 3 — Exercise extremes and seed linearity: Numerical stability must hold in both the primal and derivative path.
extremes=jnp.array([-1000.,0.,1000.])
# Vectorize across the batch dimension with `jax.vmap` (`extreme_values`).
extreme_values=jax.vmap(stable_softplus)(extremes)
extreme_gradients=jax.vmap(jax.grad(stable_softplus))(extremes)
assert np.isfinite(extreme_values).all() and np.isfinite(extreme_gradients).all()
np.testing.assert_allclose(extreme_values,[0.,np.log(2.),1000.],atol=1e-12)
with np.errstate(over='ignore'):
# Transfer the device result to the host for an independent NumPy comparison.
naive=np.log1p(np.exp(np.asarray(extremes)))
assert not np.isfinite(naive[-1])
_,pullback=jax.vjp(stable_logsumexp,x)
np.testing.assert_allclose(pullback(2.0-0.3)[0],2*pullback(1.0)[0]-pullback(0.3)[0],rtol=1e-12)
print("Stable extreme values:",np.asarray(extreme_values),"naive positive extreme:",naive[-1])Numerical stability must hold in both the primal and derivative path. Backward seed linearity is a contract of a true pullback, not an optional aesthetic property.
Explain the APIs in this step
jnp.array(values, dtype=...): Constructs a JAX array from Python/NumPy values.jax.vmap(fn, in_axes=..., out_axes=...): Vectorizes a single-example function across mapped array axes without a Python loop.jax.grad(loss_fn)(params, ...): Transforms a scalar-output function into one returning the gradient PyTree with the same structure asparams.np.isfinite(x): Returns a NumPy boolean array marking elements that are neitherNaNnor infinite.
View the cumulative build in main.py
These blocks append in order to your separate practice file.
# Step 1 — Write stable primal functions and exact derivative contracts
import jax
# Enable 64-bit arithmetic before importing jax.numpy or creating arrays.
jax.config.update("jax_enable_x64", True)
import jax.numpy as jnp
import numpy as np
# Register custom derivative rule on `stable_softplus(x)`:
@jax.custom_jvp
def stable_softplus(x):
return jnp.logaddexp(0.0,x)
@stable_softplus.defjvp
def softplus_jvp(primals,tangents):
x,=primals
dx,=tangents
return stable_softplus(x),jax.nn.sigmoid(x)*dx
# Register custom derivative rule on `stable_logsumexp(x)`:
@jax.custom_vjp
def stable_logsumexp(x):
return jax.scipy.special.logsumexp(x)
def logsumexp_fwd(x):
value=stable_logsumexp(x)
weights=jax.nn.softmax(x)
return value,weights
def logsumexp_bwd(weights,cotangent):
return (cotangent*weights,)
stable_logsumexp.defvjp(logsumexp_fwd,logsumexp_bwd)
# Step 2 — Check first and second derivatives independently
points=jnp.array([-4.,0.,3.])
# Vectorize across the batch dimension with `jax.vmap` (`values`).
values=jax.vmap(stable_softplus)(points)
first=jax.vmap(jax.grad(stable_softplus))(points)
second=jax.vmap(jax.grad(jax.grad(stable_softplus)))(points)
# Transfer the device result to the host for an independent NumPy comparison.
reference_first=1/(1+np.exp(-np.asarray(points)))
reference_second=reference_first*(1-reference_first)
# Transfer the device result to the host for an independent NumPy comparison.
np.testing.assert_allclose(values,np.logaddexp(0,np.asarray(points)),rtol=1e-12)
np.testing.assert_allclose(first,reference_first,rtol=1e-12)
np.testing.assert_allclose(second,reference_second,rtol=1e-12)
x=jnp.array([-0.8,0.4,1.1])
# Transfer the device result to the host for an independent NumPy comparison.
# Accumulate the next contribution into `weights`.
weights=np.exp(np.asarray(x)-np.max(np.asarray(x)))
weights/=weights.sum()
hessian_reference=np.diag(weights)-np.outer(weights,weights)
np.testing.assert_allclose(jax.grad(stable_logsumexp)(x),weights,rtol=1e-12)
np.testing.assert_allclose(jax.jacrev(jax.grad(stable_logsumexp))(x),hessian_reference,rtol=1e-12,atol=1e-12)
print("Softplus first/second:",np.asarray(first),np.asarray(second))
print("Logsumexp gradient:",weights)
# Step 3 — Exercise extremes and seed linearity: Numerical stability must hold in both the primal and derivative path.
extremes=jnp.array([-1000.,0.,1000.])
# Vectorize across the batch dimension with `jax.vmap` (`extreme_values`).
extreme_values=jax.vmap(stable_softplus)(extremes)
extreme_gradients=jax.vmap(jax.grad(stable_softplus))(extremes)
assert np.isfinite(extreme_values).all() and np.isfinite(extreme_gradients).all()
np.testing.assert_allclose(extreme_values,[0.,np.log(2.),1000.],atol=1e-12)
with np.errstate(over='ignore'):
# Transfer the device result to the host for an independent NumPy comparison.
naive=np.log1p(np.exp(np.asarray(extremes)))
assert not np.isfinite(naive[-1])
_,pullback=jax.vjp(stable_logsumexp,x)
np.testing.assert_allclose(pullback(2.0-0.3)[0],2*pullback(1.0)[0]-pullback(0.3)[0],rtol=1e-12)
print("Stable extreme values:",np.asarray(extreme_values),"naive positive extreme:",naive[-1])Run the example
python3 phases/14-internals/03-custom-derivative-rules/code/main.pyAbout this command
This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.
Inspect the complete assembled example
# Step 1 — Write stable primal functions and exact derivative contracts
import jax
# Enable 64-bit arithmetic before importing jax.numpy or creating arrays.
jax.config.update("jax_enable_x64", True)
import jax.numpy as jnp
import numpy as np
# Register custom derivative rule on `stable_softplus(x)`:
@jax.custom_jvp
def stable_softplus(x):
return jnp.logaddexp(0.0,x)
@stable_softplus.defjvp
def softplus_jvp(primals,tangents):
x,=primals
dx,=tangents
return stable_softplus(x),jax.nn.sigmoid(x)*dx
# Register custom derivative rule on `stable_logsumexp(x)`:
@jax.custom_vjp
def stable_logsumexp(x):
return jax.scipy.special.logsumexp(x)
def logsumexp_fwd(x):
value=stable_logsumexp(x)
weights=jax.nn.softmax(x)
return value,weights
def logsumexp_bwd(weights,cotangent):
return (cotangent*weights,)
stable_logsumexp.defvjp(logsumexp_fwd,logsumexp_bwd)
# Step 2 — Check first and second derivatives independently
points=jnp.array([-4.,0.,3.])
# Vectorize across the batch dimension with `jax.vmap` (`values`).
values=jax.vmap(stable_softplus)(points)
first=jax.vmap(jax.grad(stable_softplus))(points)
second=jax.vmap(jax.grad(jax.grad(stable_softplus)))(points)
# Transfer the device result to the host for an independent NumPy comparison.
reference_first=1/(1+np.exp(-np.asarray(points)))
reference_second=reference_first*(1-reference_first)
# Transfer the device result to the host for an independent NumPy comparison.
np.testing.assert_allclose(values,np.logaddexp(0,np.asarray(points)),rtol=1e-12)
np.testing.assert_allclose(first,reference_first,rtol=1e-12)
np.testing.assert_allclose(second,reference_second,rtol=1e-12)
x=jnp.array([-0.8,0.4,1.1])
# Transfer the device result to the host for an independent NumPy comparison.
# Accumulate the next contribution into `weights`.
weights=np.exp(np.asarray(x)-np.max(np.asarray(x)))
weights/=weights.sum()
hessian_reference=np.diag(weights)-np.outer(weights,weights)
np.testing.assert_allclose(jax.grad(stable_logsumexp)(x),weights,rtol=1e-12)
np.testing.assert_allclose(jax.jacrev(jax.grad(stable_logsumexp))(x),hessian_reference,rtol=1e-12,atol=1e-12)
print("Softplus first/second:",np.asarray(first),np.asarray(second))
print("Logsumexp gradient:",weights)
# Step 3 — Exercise extremes and seed linearity: Numerical stability must hold in both the primal and derivative path.
extremes=jnp.array([-1000.,0.,1000.])
# Vectorize across the batch dimension with `jax.vmap` (`extreme_values`).
extreme_values=jax.vmap(stable_softplus)(extremes)
extreme_gradients=jax.vmap(jax.grad(stable_softplus))(extremes)
assert np.isfinite(extreme_values).all() and np.isfinite(extreme_gradients).all()
np.testing.assert_allclose(extreme_values,[0.,np.log(2.),1000.],atol=1e-12)
with np.errstate(over='ignore'):
# Transfer the device result to the host for an independent NumPy comparison.
naive=np.log1p(np.exp(np.asarray(extremes)))
assert not np.isfinite(naive[-1])
_,pullback=jax.vjp(stable_logsumexp,x)
np.testing.assert_allclose(pullback(2.0-0.3)[0],2*pullback(1.0)[0]-pullback(0.3)[0],rtol=1e-12)
print("Stable extreme values:",np.asarray(extreme_values),"naive positive extreme:",naive[-1])Expected result: Softplus derivatives at zero are 0.5 and 0.25. Stable values at [-1000, 0, 1000] are [0, log(2), 1000]; the naive positive extreme overflows. The custom reverse Hessian matches diag(p)-p p-transpose.
Recorded output and runtime details
Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.
Softplus first/second: [0.01798621 0.5 0.95257413] [0.01766271 0.25 0.04517666]
Logsumexp gradient: [0.09085944 0.30166396 0.60747661]
Stable extreme values: [0.00000000e+00 6.93147181e-01 1.00000000e+03] naive positive extreme: inf
Softplus first/second: [0.01798621 0.5 0.95257413] [0.01766271 0.25 0.04517666]
Logsumexp gradient: [0.09085944 0.30166396 0.60747661]
Stable extreme values: [0.00000000e+00 6.93147181e-01 1.00000000e+03] naive positive extreme: inf
Wrong/correct/finite-difference derivative: 0.29934383005622606 0.5986876601124521 0.5986876601138391
Expected custom_vjp forward-mode boundary: can't apply forward-mode autodiff (jvp) to a custom_vjp function.
Ordinary JAX common-shift JVP: 1.0
Changed-size gradient and cotangent scaling verified
Stable shift invariance and zero common-direction curvature verified
Nonlinear seed rule rejected by its contract; repaired JVP is linear
PASS: internals-03
Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.
Expose a wrong but linear rule
# Experiment — Expose a wrong but linear rule
# Register custom derivative rule on `wrong_softplus(z)`:
@jax.custom_jvp
def wrong_softplus(z):
return jnp.logaddexp(0.0,z)
@wrong_softplus.defjvp
def wrong_rule(primals,tangents):
z,=primals
dz,=tangents
return wrong_softplus(z),0.5*jax.nn.sigmoid(z)*dz
probe=0.4
eps=1e-5
finite=(np.logaddexp(0,probe+eps)-np.logaddexp(0,probe-eps))/(2*eps)
wrong=float(jax.grad(wrong_softplus)(probe))
right=float(jax.grad(stable_softplus)(probe))
assert abs(wrong-finite)>0.2
np.testing.assert_allclose(right,finite,rtol=1e-9)
print("Wrong/correct/finite-difference derivative:",wrong,right,finite)Expected: Wrong derivative approximately 0.29934; correct and finite-difference derivatives approximately 0.59869.
The wrong tangent expression is linear, so transposition can still work. Only an independent mathematical or numerical oracle detects its incorrect coefficient.
Exercise the custom VJP boundary
# Experiment — Exercise the custom VJP boundary
rejected=False
try:
jax.jvp(stable_logsumexp,(x,),(jnp.ones_like(x),))
except TypeError as error:
rejected=True
print("Expected custom_vjp forward-mode boundary:",str(error).splitlines()[0])
assert rejected
ordinary=lambda z:jax.scipy.special.logsumexp(z)
ordinary_direction=jax.jvp(ordinary,(x,),(jnp.ones_like(x),))[1]
np.testing.assert_allclose(ordinary_direction,1.0,atol=1e-12)
print("Ordinary JAX common-shift JVP:",float(ordinary_direction))Expected: The custom_vjp direct JVP is rejected; the ordinary function gives directional derivative 1.
Adding the same constant to all logits shifts log-sum-exp by that constant. The mathematical derivative exists; the limitation belongs to the custom_vjp transformation interface.
Make it yours
Where to put your code: Modify your own main.py, or use the Make it yours cell in your downloaded notebook. Run your attempt in the declared environment and verify your prediction before opening the reference solution.
Check the custom log-sum-exp gradient on against an independent stable softmax and central differences along two directions. Keep the output cotangent explicit.
Show a writing guide and starter scaffold
How to write this exercise code
Learner-only inputs, outputs, checks and hints
Inputs
- Logits (-3, 0.2, 2, 5), explicit scalar output cotangent, and two directions.
Outputs
- Custom-gradient result, stable softmax reference, and centered directional differences.
Invariants to check
- The custom gradient scales by the explicit cotangent.
- Softmax reference is stable under large logits.
Hints
- Subtract the maximum before exponentiating.
- Use the same cotangent for autodiff and independent directional checks.
import numpy as np
import jax
import jax.numpy as jnp
def check_custom_logsumexp(logits, cotangent, directions, epsilon):
# TODO: compare custom VJP with stable softmax and directional differences.
raise NotImplementedError("Audit the explicit cotangent and two independent directions")Reveal the reference solution
# Exercise solution
changed=jnp.array([-3.,0.2,2.,5.])
# Transfer the device result to the host for an independent NumPy comparison.
# Accumulate the next contribution into `p`.
host=np.asarray(changed)
p=np.exp(host-host.max())
p/=p.sum()
actual=jax.grad(stable_logsumexp)(changed)
np.testing.assert_allclose(actual,p,rtol=1e-12)
for direction in (np.array([1.,-.5,.2,0.]),np.array([0.,1.,-1.,.3])):
eps=1e-5
reference=lambda z:np.max(z)+np.log(np.exp(z-np.max(z)).sum())
fd=(reference(host+eps*direction)-reference(host-eps*direction))/(2*eps)
# Transfer the device result to the host for an independent NumPy comparison.
np.testing.assert_allclose(np.dot(np.asarray(actual),direction),fd,rtol=1e-8,atol=1e-10)
np.testing.assert_allclose(jax.vjp(stable_logsumexp,changed)[1](2.5)[0],2.5*p,rtol=1e-12)
print("Changed-size gradient and cotangent scaling verified")Check common-shift invariance of curvature
Practice
Add a common constant of to every logit. Verify the value shift, unchanged gradient, and Hessian row sums.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- A stable log-sum-exp custom derivative, a logit vector, and a common shift of 1000.
Outputs
- Value shift, gradient comparison, and Hessian row sums before/after shift.
Invariants to check
- The value increases by the common shift.
- Gradient stays invariant and Hessian has near-zero row sums.
Hints
- Use centered logits for numerical stability.
- Compute Hessian row sums from derivatives, not from a stored expected answer.
import jax
import jax.numpy as jnp
def check_shift_curvature(logsumexp, logits, shift):
# TODO: compare values, gradients, and Hessian row sums under a common shift.
raise NotImplementedError("Check shift invariance of first and second derivatives")Show a hint
Use stable references and avoid direct exponentiation of the shifted logits.
Reveal reference solution and reasoning
# Check common-shift invariance of curvature (Practice): The common shift leaves probabilities unchanged.
shift=1000.0
np.testing.assert_allclose(stable_logsumexp(x+shift)-stable_logsumexp(x),shift,rtol=1e-12)
np.testing.assert_allclose(jax.grad(stable_logsumexp)(x+shift),jax.grad(stable_logsumexp)(x),rtol=1e-12)
hess=jax.jacrev(jax.grad(stable_logsumexp))(x+shift)
np.testing.assert_allclose(jnp.sum(hess,axis=1),0.,atol=1e-12)
print("Stable shift invariance and zero common-direction curvature verified")The common shift leaves probabilities unchanged. Zero curvature in that direction is a structural identity, not a numerical accident.
Audit a nonlinear seed rule before registering it
Challenge
A proposed JVP returns . Show explicitly why it cannot be the derivative map, then repair it and test a mixed linear combination.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- A proposed JVP rule returning sigmoid(x)*dx**2 and a repaired tangent-linear rule.
Outputs
- A mixed-seed counterexample to additivity and a repaired-rule check.
Invariants to check
- A derivative is linear in the tangent at fixed primal input.
- The repaired rule passes mixed linear-combination checks.
Hints
- Compare the rule at dx1, dx2, and dx1+dx2.
- Keep the primal input fixed in each additivity test.
import jax.numpy as jnp
def audit_jvp_rule(primal, dx1, dx2):
# TODO: expose nonlinearity in the proposed seed rule, then repair it.
raise NotImplementedError("Require additivity and homogeneity in every tangent seed")Show a hint
At fixed x, compare the response to twice a tangent with twice the response.
Reveal reference solution and reasoning
# Audit a nonlinear seed rule before registering it (Challenge)
coefficient=float(jax.nn.sigmoid(0.4))
bad=lambda tangent:coefficient*tangent*tangent
assert not np.isclose(bad(2.0),2*bad(1.0))
good=lambda tangent:jax.jvp(stable_softplus,(jnp.array(0.4),),(jnp.array(tangent),))[1]
np.testing.assert_allclose(good(2*.7-.3),2*good(.7)-good(.3),rtol=1e-12)
print("Nonlinear seed rule rejected by its contract; repaired JVP is linear")Linearity follows from the definition of a derivative at a fixed primal point. API acceptance is not a substitute for this mathematical check.
Diagnose the result
If first derivatives pass but second derivatives fail, inspect whether residuals or derivative coefficients have been detached. If extreme values are nonfinite, inspect both primal and derivative expressions. If a custom VJP direct JVP fails, treat that as an interface boundary. If a plausible custom gradient differs from finite differences, do not relabel it exact without a valid derivation.
Carry forward
- Keep the primal stable before customizing its derivative
- A custom JVP must remain linear in its tangent
- Choose residuals for a custom reverse pass
- Higher-order behavior is part of the promise
- Custom reverse mode has a transformation boundary
- An intentionally different gradient needs a different claim
Keep your evidence
Stable extreme inputs, analytic slopes/Hessians, seed linearity, independent finite differences and a deliberately wrong-rule diagnosis. Keep the environment, observed outputs and your explanation of the figure.
Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.
Saved in this browser. Export or back up your portfolio anytime in My learning.
Reference execution: CPU validation only. This worked reference is validated on CPU. Accelerator compatibility requires a separate source review and target execution receipt. Use the execution matrix to review device selection and data ownership before adapting a separate target program. Execution matrix and target qualification.