Phase 11 / Probabilistic modeling
Distributions and sampling
Revision v0.18.1 · Updated since your last visit. Review the changes; your progress is saved.
Lesson · about 90 minutes · CPU practice
Distinguish samples, density, variance and standard error Check normal moments against independent analytic expectations
Phase study guide: preparation, milestones and project evidence
Before you start
Before this: Math & optimization
Hardware: CPU
Setup help: Python, notebook, or Cloud TPU VM
Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions require separate checks. Hardware receipts apply only to their recorded source, environment, hardware, and cases; they do not qualify every lesson or TPU configuration. See the TPU performance and hardware evidence guide. The optional Cloud TPU specialization explains target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.
Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txtpy -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txtAfter activation, run the complete worked example from that same folder:
python phases/11-probability/01-distributions-and-sampling/code/main.py.\.venv\Scripts\python.exe phases/11-probability/01-distributions-and-sampling/code/main.pyFor an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.
Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.
For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:
python -m pip install jupyterlab
python -m jupyter lab.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter labSelect the downloaded probability-01.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.
The worked reference for this lesson is CPU-validated. Some references explicitly configure logical CPU devices; setting JAX_PLATFORMS does not adapt them to a TPU. Follow the execution matrix and reviewed multi-host example before attempting a target run.
For accelerator preparation, study TPU setup, observed runtime receipts, checkpointed jobs and precision and retained traces. Current receipts cover bounded v5e1 launch, resume, and precision checks; four single-device Pallas cases; and a four-device v5e collective fixture. Each receipt is limited to its recorded source, environment, cases, and hardware. It does not qualify all lessons or TPU configurations. Cloud provisioning is optional for CPU learning.
Check: The sample mean is close to , the variance is close to , and the stable mixture log-density is approximately .
Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.
What you will be able to do
- Distinguish samples, density, variance and standard error
- Check normal moments against independent analytic expectations
- Use log-space arithmetic for an extreme mixture likelihood
- Demonstrate both key replay and independent split-key draws
The problem
A sensor model says that a future reading has mean and standard deviation . What should a finite collection of simulated readings look like, and how can we check our simulator? We will compare independent formulas with JAX samples, then deliberately underflow a likelihood calculation. This is the numerical foundation for the Bayesian model in the next lesson.
The idea
A distribution describes variability in outcomes. Averaging independent observations can reduce uncertainty in their mean without removing the spread of individual outcomes. Keep these two uncertainties on separate labels and scales.
A precise mean does not make individual outcomes predictable
For independent readings with standard deviation , the sample mean of readings has standard deviation , while a new reading still has standard deviation . The first quantity describes variation in an average across repeated samples; the second describes the data-generating spread of individual values.
The displayed standard-error curve is calculated from the independence assumption, not estimated from repeated experimental batches. Positive correlation would make too optimistic because the effective amount of independent information is smaller. Before using a narrow band, identify whether its random quantity is a mean, parameter or future observation.
Start with units, not an API
A standard deviation has the same units as a sensor reading. Variance has squared units. To change a standard-normal draw into a reading, multiply by the standard deviation and add the mean: . The expected variance is , not . This is why the scale parameter in our log-density divides the residual before squaring. We assume a strictly positive finite scale; validation belongs at a public input boundary.
One draw and a sample average answer different questions
A single future reading has standard deviation . An average of independent readings has standard error . More simulation improves an estimate of the model mean; it does not make the next sensor reading less noisy. At , the mean standard error is about . Our deterministic seeded test uses a generous five-standard-error bound and a separate variance check. A statistical tolerance can fail by chance for other seeds, so keep the seed, sample size and actual discrepancy.
Logarithms preserve ratios that densities lose
A probability density is not the probability of observing one exact real-valued reading. Density has inverse-reading units, and probability comes from integrating it over an interval or region. For example, the normal density at its peak can exceed one for a sufficiently small standard deviation without violating probability rules.
A normal log-density is a negative quadratic plus normalization terms. Summing log-densities for independent observations avoids multiplying many small values. A mixture instead adds weighted densities, so use log-sum-exp: for component log-densities and , subtracting the maximum before exponentiation preserves their relative contribution. The executed stable mixture log density is about ; direct exponentiation underflows at this scale even though the log result remains finite.
A random key is part of the experiment
Calling a random function twice with the same key and shape replays the same values in this tested environment. It does not create two independent samples. Split the key before assigning it to separate draws. NumPy and JAX need not return matching random arrays for the same integer seed: compare distributional properties, not bit patterns across libraries. Our sample checks establish a small numerical example, not a universal test of a random-number generator.
A precise mean does not remove observation noise
Predict before looking: Which band shrinks when we increase the number of independent observations?
Read the figure
The horizontal axis is independent sample count on a logarithmic scale, so equal horizontal steps multiply sample size by the same ratio. The vertical axis is standard deviation in sensor-reading units. The individual-reading curve stays at 3; the sample-mean curve is , reaching 0.3 at . These are analytic standard deviations under independence, not observed confidence-interval coverage.
Connect it to the computation
At , averaging reduces uncertainty about the mean by a factor of ten but does not shrink the spread of the next reading. The seeded sample check reports mean 2.0067 and variance 9.0353, close to the model values 2 and 9 for this finite draw. Correlated draws would not follow the same standard-error curve without accounting for dependence.
Figure data and execution details
Open full-size figureDownload data and provenance
The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.
Your build: Create a separate main.py in your practice folder. Append each block in order and run python main.py in the lesson environment. The downloaded script is the complete reference, so keep it separate from your attempt.
1 Define a density and its contract
Create main.py in the activated CPU environment. Add the imports and the scalar normal log-density. A scale is a positive standard deviation, not a variance.
# Step 1 — 1. Define a density and its contract
import math
import numpy as np
import jax
import jax.numpy as jnp
from jax.scipy.special import logsumexp
def normal_logpdf(value, loc, scale):
return -0.5*((value-loc)/scale)**2 - jnp.log(scale) - 0.5*jnp.log(2*jnp.pi)The normalization term matters when comparing different scales; the squared residual alone is not a density.
2 Sample with explicit ownership
Append the sampler. Split one parent key before drawing two samples; retain each array so replay and variation can be checked.
# Step 2 — 2. Sample with explicit ownership
# Create or split explicit PRNG key(s) (`(key_a, key_b)`) for reproducible randomness.
key_a, key_b = jax.random.split(jax.random.key(23))
# Sample deterministic random values into `a` using an explicit PRNG key.
a = 2. + 3.*jax.random.normal(key_a, (20000,))
# Sample deterministic random values into `b` using an explicit PRNG key.
b = 2. + 3.*jax.random.normal(key_b, (20000,))
expected_mean, expected_variance = 2., 9.
assert abs(float(a.mean())-expected_mean) < 5*3/math.sqrt(a.size)
assert abs(float(a.var())-expected_variance) < .4
assert not jnp.array_equal(a,b)
assert jnp.array_equal(a, 2.+3.*jax.random.normal(key_a,(20000,)))The broad mean bound is five analytic standard errors for this independent normal sample; it is not a guarantee for arbitrary Monte Carlo samples.
Explain the APIs in this step
jax.random.split(key, num=...): Derives new keys for independent random operations; using the same key again repeats its random stream.jax.random.key(seed): Creates a typed explicit PRNG key from a seed.jax.random.normal(key, shape, dtype=...): Draws normal random values using an explicit PRNG key.array.mean(axis=..., keepdims=...): Averages values along the selected axis; that axis is removed unlesskeepdims=True.
3 Compare an independent calculation
Append the independent scalar reference and stable mixture calculation, then run python main.py. Inspect the density, mean and variance.
# Step 3 — 3. Compare an independent calculation
actual = normal_logpdf(jnp.array([2.,5.]),2.,3.)
reference = [-math.log(3*math.sqrt(2*math.pi)), -.5-math.log(3*math.sqrt(2*math.pi))]
np.testing.assert_allclose(actual,reference,rtol=1e-6)
components = jnp.array([-1000.,-1001.])
stable = logsumexp(components)-jnp.log(2.)
assert jnp.isfinite(stable)
assert jnp.isneginf(jnp.log(jnp.exp(components).mean()))
print('sample mean, variance:',float(a.mean()),float(a.var()))
print('stable equal-mixture log density:',float(stable))A log density near negative one thousand remains representable even when exponentiating it underflows in float32.
Explain the APIs in this step
jnp.array(values, dtype=...): Constructs a JAX array from Python/NumPy values.np.testing.assert_allclose(actual, desired, rtol=..., atol=...): Checks a host-side numerical expectation and raisesAssertionErrorwhen values differ beyond tolerance.jnp.isfinite(x): Returns a JAX boolean array marking elements that are neitherNaNnor infinite.array.mean(axis=..., keepdims=...): Averages values along the selected axis; that axis is removed unlesskeepdims=True.
View the cumulative build in main.py
These blocks append in order to your separate practice file.
# Step 1 — 1. Define a density and its contract
import math
import numpy as np
import jax
import jax.numpy as jnp
from jax.scipy.special import logsumexp
def normal_logpdf(value, loc, scale):
return -0.5*((value-loc)/scale)**2 - jnp.log(scale) - 0.5*jnp.log(2*jnp.pi)
# Step 2 — 2. Sample with explicit ownership
# Create or split explicit PRNG key(s) (`(key_a, key_b)`) for reproducible randomness.
key_a, key_b = jax.random.split(jax.random.key(23))
# Sample deterministic random values into `a` using an explicit PRNG key.
a = 2. + 3.*jax.random.normal(key_a, (20000,))
# Sample deterministic random values into `b` using an explicit PRNG key.
b = 2. + 3.*jax.random.normal(key_b, (20000,))
expected_mean, expected_variance = 2., 9.
assert abs(float(a.mean())-expected_mean) < 5*3/math.sqrt(a.size)
assert abs(float(a.var())-expected_variance) < .4
assert not jnp.array_equal(a,b)
assert jnp.array_equal(a, 2.+3.*jax.random.normal(key_a,(20000,)))
# Step 3 — 3. Compare an independent calculation
actual = normal_logpdf(jnp.array([2.,5.]),2.,3.)
reference = [-math.log(3*math.sqrt(2*math.pi)), -.5-math.log(3*math.sqrt(2*math.pi))]
np.testing.assert_allclose(actual,reference,rtol=1e-6)
components = jnp.array([-1000.,-1001.])
stable = logsumexp(components)-jnp.log(2.)
assert jnp.isfinite(stable)
assert jnp.isneginf(jnp.log(jnp.exp(components).mean()))
print('sample mean, variance:',float(a.mean()),float(a.var()))
print('stable equal-mixture log density:',float(stable))Run the example
python3 phases/11-probability/01-distributions-and-sampling/code/main.pyAbout this command
This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.
Inspect the complete assembled example
# Step 1 — 1. Define a density and its contract
import math
import numpy as np
import jax
import jax.numpy as jnp
from jax.scipy.special import logsumexp
def normal_logpdf(value, loc, scale):
return -0.5*((value-loc)/scale)**2 - jnp.log(scale) - 0.5*jnp.log(2*jnp.pi)
# Step 2 — 2. Sample with explicit ownership
# Create or split explicit PRNG key(s) (`(key_a, key_b)`) for reproducible randomness.
key_a, key_b = jax.random.split(jax.random.key(23))
# Sample deterministic random values into `a` using an explicit PRNG key.
a = 2. + 3.*jax.random.normal(key_a, (20000,))
# Sample deterministic random values into `b` using an explicit PRNG key.
b = 2. + 3.*jax.random.normal(key_b, (20000,))
expected_mean, expected_variance = 2., 9.
assert abs(float(a.mean())-expected_mean) < 5*3/math.sqrt(a.size)
assert abs(float(a.var())-expected_variance) < .4
assert not jnp.array_equal(a,b)
assert jnp.array_equal(a, 2.+3.*jax.random.normal(key_a,(20000,)))
# Step 3 — 3. Compare an independent calculation
actual = normal_logpdf(jnp.array([2.,5.]),2.,3.)
reference = [-math.log(3*math.sqrt(2*math.pi)), -.5-math.log(3*math.sqrt(2*math.pi))]
np.testing.assert_allclose(actual,reference,rtol=1e-6)
components = jnp.array([-1000.,-1001.])
stable = logsumexp(components)-jnp.log(2.)
assert jnp.isfinite(stable)
assert jnp.isneginf(jnp.log(jnp.exp(components).mean()))
print('sample mean, variance:',float(a.mean()),float(a.var()))
print('stable equal-mixture log density:',float(stable))Expected result: The sample mean is close to , the variance is close to , and the stable mixture log-density is approximately .
Recorded output and runtime details
Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.
sample mean, variance: 2.0067408084869385 9.035282135009766
stable equal-mixture log density: -1000.3799438476562
sample mean, variance: 2.0067408084869385 9.035282135009766
stable equal-mixture log density: -1000.3799438476562
peak density: 3.9894230365753174
-5.0 -2.379885673522949
PASS: probability-01
Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.
Density can exceed one
# Experiment — Density can exceed one
peak = jnp.exp(normal_logpdf(0.,0.,.1))
assert peak > 1
print("peak density:",float(peak))Expected: The peak density is approximately .
A narrow continuous density can exceed one; its area, not its height at a point, sums to one.
Separate sum and mixture
# Experiment — Separate sum and mixture
values = jnp.array([-2.,-3.])
independent_log = values.sum()
mixture_log = logsumexp(values)-jnp.log(2.)
assert jnp.allclose(independent_log,-5.)
assert mixture_log > -3.
print(float(independent_log),float(mixture_log))Expected: The independent joint log likelihood is ; the mixture log density is approximately .
Independence multiplies probabilities, while alternative mixture components add weighted probabilities.
Make it yours
Where to put your code: Modify your own main.py, or use the Make it yours cell in your downloaded notebook. Run your attempt in the declared environment and verify your prediction before opening the reference solution.
Change the sensor to mean , standard deviation , and draw readings from a new key. Predict the variance and the standard error of the sample mean, then verify both moments.
Show a writing guide and starter scaffold
How to write this exercise code
Learner-only inputs, outputs, checks and hints
Inputs
- A seed for a fresh observation key; mean and standard deviation of the sensor.
Outputs
- One reading vector, its sample variance, and the standard error of its sample mean.
Invariants to check
- The input key is not reused for another random draw.
- The reported standard error uses the number of readings, not the variance alone.
Hints
- Split a key before sampling.
- For independent readings, standard error scales as standard deviation divided by the square root of the sample count.
- Compare empirical moments with finite-sample tolerances rather than exact equality.
def compare_sample_moments(key, mean, scale, count):
"""Return readings, sample variance, and standard error of the mean."""
# TODO: split the key, draw readings, and compute both summaries.
raise NotImplementedErrorReveal the reference solution
# Exercise solution
# Create or split explicit PRNG key(s) (`changed`) for reproducible randomness.
changed = -1.+.5*jax.random.normal(jax.random.key(51),(30000,))
assert abs(float(changed.mean())+1.) < 5*.5/math.sqrt(30000)
assert abs(float(changed.var())-.25) < .015Detect a missing normalization
Transfer / diagnosis
Compare scales and at their common mean. Show why a residual-only objective incorrectly ties them.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- The shared mean and two normal scales, 1 and 2.
Outputs
- The two log densities at the common mean and the change in the normalized density across scales.
Invariants to check
- The residual-only term is identical at the shared mean, while the full log densities differ because scale changes the normalizer.
Hints
- Evaluate both normalized log densities at the same zero-residual point.
- Compare the full expression with the residual-only term.
import jax.numpy as jnp
def compare_normal_scales_at_common_mean(normal_logpdf):
# TODO: Evaluate scales one and two at their common mean and expose the residual-only tie.
raise NotImplementedErrorShow a hint
The residual is zero for both; retain .
Reveal reference solution and reasoning
# Detect a missing normalization (Transfer / diagnosis): Scale changes density even with identical zero residuals.
scores = normal_logpdf(0.,0.,jnp.array([1.,2.]))
assert jnp.allclose(scores[0]-scores[1],jnp.log(2.))Scale changes density even with identical zero residuals.
Demonstrate key ownership
Transfer / diagnosis
Produce two separately owned standard-normal draws and show replay of the first. Explain why replay alone does not test independence.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- An explicit JAX key, two child-work labels, and a changed seed.
Outputs
- Two reproducible but independent random arrays.
Invariants to check
- Reusing a key replays its random values; splitting creates independent child streams.
Hints
- Split once for each random operation. Verify replay and changed-seed behavior separately.
def check_key_ownership(key, seed):
"""Return two draws and a replay/changed-seed comparison."""
# TODO: allocate child keys and record the random-stream checks.
raise NotImplementedErrorShow a hint
Split before use; compare arrays with the same shape.
Reveal reference solution and reasoning
# Demonstrate key ownership (Transfer / diagnosis)
# Create or split explicit PRNG key(s) (`(left, right)`) for reproducible randomness.
left,right = jax.random.split(jax.random.key(17))
# Sample deterministic random values into `first` using an explicit PRNG key.
first = jax.random.normal(left,(32,))
# Sample deterministic random values into `second` using an explicit PRNG key.
second = jax.random.normal(right,(32,))
assert jnp.array_equal(first,jax.random.normal(left,(32,)))
assert not jnp.array_equal(first,second)Different split keys avoid accidental replay; one differing pair does not prove every statistical property of a generator.
Diagnose the result
If the simulated variance is near three instead of nine, inspect whether you multiplied by sqrt when the stated scale was three. If two supposedly independent arrays match exactly, inspect key reuse before changing the distribution. If a mixture likelihood becomes negative infinity, compare the log-sum-exp result before widening dtype.
Carry forward
- A density is not a point probability.
- Moment checks need sample size, units and a justified tolerance.
- Log-space arithmetic and explicit keys make likelihood experiments inspectable.
Keep your evidence
Seeded arrays, independent density values, moment discrepancies and underflow diagnosis. Keep the environment, observed outputs and your explanation of the figure.
Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.
Saved in this browser. Export or back up your portfolio anytime in My learning.
Reference execution: CPU validation only. This worked reference is validated on CPU. Accelerator compatibility requires a separate source review and target execution receipt. Use the execution matrix to review device selection and data ownership before adapting a separate target program. Execution matrix and target qualification.