Domain Specialization · Local CPULaptop CPULesson 1 of 4 in Phase 11

Phase 11 / Probabilistic modeling

Distributions and sampling

Revision v0.18.1 ·

Lesson · about 90 minutes · CPU practice

Distinguish samples, density, variance and standard error Check normal moments against independent analytic expectations

Phase study guide: preparation, milestones and project evidence

Before you start

Before this: Math & optimization
Hardware: CPU

Setup help: Python, notebook, or Cloud TPU VM

Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions require separate checks. Hardware receipts apply only to their recorded source, environment, hardware, and cases; they do not qualify every lesson or TPU configuration. See the TPU performance and hardware evidence guide. The optional Cloud TPU specialization explains target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.

Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.

macOS / Linux terminal · repository folder
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txt

PowerShell · repository folder
py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txt

After activation, run the complete worked example from that same folder:

macOS / Linux terminal · activated environment
python phases/11-probability/01-distributions-and-sampling/code/main.py

PowerShell command · repository folder
.\.venv\Scripts\python.exe phases/11-probability/01-distributions-and-sampling/code/main.py

For an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.

Check: The sample mean is close to 22, the variance is close to 99, and the stable mixture log-density is approximately −1000.38-1000.38.

Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.

Stage 1 of 5 · Understand the MechanismCore mental model, equations & visual walkthrough

What you will be able to do

  • Distinguish samples, density, variance and standard error
  • Check normal moments against independent analytic expectations
  • Use log-space arithmetic for an extreme mixture likelihood
  • Demonstrate both key replay and independent split-key draws

The problem

A sensor model says that a future reading has mean 22 and standard deviation 33. What should a finite collection of simulated readings look like, and how can we check our simulator? We will compare independent formulas with JAX samples, then deliberately underflow a likelihood calculation. This is the numerical foundation for the Bayesian model in the next lesson.

The idea

A distribution describes variability in outcomes. Averaging independent observations can reduce uncertainty in their mean without removing the spread of individual outcomes. Keep these two uncertainties on separate labels and scales.

A precise mean does not make individual outcomes predictable

For independent readings with standard deviation 33, the sample mean of 100100 readings has standard deviation 3/100=0.33/\sqrt{100}=0.3, while a new reading still has standard deviation 33. The first quantity describes variation in an average across repeated samples; the second describes the data-generating spread of individual values.

The displayed standard-error curve is calculated from the independence assumption, not estimated from repeated experimental batches. Positive correlation would make 2/N2/\sqrt{N} too optimistic because the effective amount of independent information is smaller. Before using a narrow band, identify whether its random quantity is a mean, parameter or future observation.

Start with units, not an API

A standard deviation has the same units as a sensor reading. Variance has squared units. To change a standard-normal draw zz into a reading, multiply by the standard deviation and add the mean: x=2+3zx=2+3z. The expected variance is 99, not 33. This is why the scale parameter in our log-density divides the residual before squaring. We assume a strictly positive finite scale; validation belongs at a public input boundary.

p(x∣μ,σ)=1σ2πexp⁡ ⁣[−12(x−μσ)2],σ>0p(x\mid\mu,\sigma)=\frac{1}{\sigma\sqrt{2\pi}}\exp\!\left[-\frac12\left(\frac{x-\mu}{\sigma}\right)^2\right],\quad \sigma>0

One draw and a sample average answer different questions

A single future reading has standard deviation 33. An average of NN independent readings has standard error 3/N3/\sqrt{N}. More simulation improves an estimate of the model mean; it does not make the next sensor reading less noisy. At N=20000N=20000, the mean standard error is about 0.02120.0212. Our deterministic seeded test uses a generous five-standard-error bound and a separate variance check. A statistical tolerance can fail by chance for other seeds, so keep the seed, sample size and actual discrepancy.

Logarithms preserve ratios that densities lose

A probability density is not the probability of observing one exact real-valued reading. Density has inverse-reading units, and probability comes from integrating it over an interval or region. For example, the normal density at its peak can exceed one for a sufficiently small standard deviation without violating probability rules.

A normal log-density is a negative quadratic plus normalization terms. Summing log-densities for independent observations avoids multiplying many small values. A mixture instead adds weighted densities, so use log-sum-exp: for component log-densities −1000-1000 and −1001-1001, subtracting the maximum before exponentiation preserves their relative contribution. The executed stable mixture log density is about −1000.380-1000.380; direct exponentiation underflows at this scale even though the log result remains finite.

log⁡ ⁣(ea+eb2)=m+log⁡(ea−m+eb−m)−log⁡2,m=max⁡(a,b)\log\!\left(\frac{e^a+e^b}{2}\right)=m+\log(e^{a-m}+e^{b-m})-\log 2,\quad m=\max(a,b)

A random key is part of the experiment

Calling a random function twice with the same key and shape replays the same values in this tested environment. It does not create two independent samples. Split the key before assigning it to separate draws. NumPy and JAX need not return matching random arrays for the same integer seed: compare distributional properties, not bit patterns across libraries. Our sample checks establish a small numerical example, not a universal test of a random-number generator.

Read the figure

The horizontal axis is independent sample count NN on a logarithmic scale, so equal horizontal steps multiply sample size by the same ratio. The vertical axis is standard deviation in sensor-reading units. The individual-reading curve stays at 3; the sample-mean curve is 3/N3/\sqrt{N}, reaching 0.3 at N=100N=100. These are analytic standard deviations under independence, not observed confidence-interval coverage.

Connect it to the computation

At N=100N=100, averaging reduces uncertainty about the mean by a factor of ten but does not shrink the spread of the next reading. The seeded sample check reports mean 2.0067 and variance 9.0353, close to the model values 2 and 9 for this finite draw. Correlated draws would not follow the same standard-error curve without accounting for dependence.

Figure data and execution details

Recorded environment: JAX 0.9.2 · 1 CPU device

The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.

Stage 2 of 5 · Build & Run the ReferenceStep-by-step code construction & verified execution output

Your build: Create a separate main.py in your practice folder. Append each block in order and run python main.py in the lesson environment. The downloaded script is the complete reference, so keep it separate from your attempt.

1 Define a density and its contract

Create main.py in the activated CPU environment. Add the imports and the scalar normal log-density. A scale is a positive standard deviation, not a variance.

main.py · append build step 1
# Step 1 — 1. Define a density and its contract
import math
import numpy as np
import jax
import jax.numpy as jnp
from jax.scipy.special import logsumexp

def normal_logpdf(value, loc, scale):
    return -0.5*((value-loc)/scale)**2 - jnp.log(scale) - 0.5*jnp.log(2*jnp.pi)

The normalization term matters when comparing different scales; the squared residual alone is not a density.

2 Sample with explicit ownership

Append the sampler. Split one parent key before drawing two samples; retain each array so replay and variation can be checked.

main.py · append build step 2
# Step 2 — 2. Sample with explicit ownership
# Create or split explicit PRNG key(s) (`(key_a, key_b)`) for reproducible randomness.
key_a, key_b = jax.random.split(jax.random.key(23))
# Sample deterministic random values into `a` using an explicit PRNG key.
a = 2. + 3.*jax.random.normal(key_a, (20000,))
# Sample deterministic random values into `b` using an explicit PRNG key.
b = 2. + 3.*jax.random.normal(key_b, (20000,))
expected_mean, expected_variance = 2., 9.
assert abs(float(a.mean())-expected_mean) < 5*3/math.sqrt(a.size)
assert abs(float(a.var())-expected_variance) < .4
assert not jnp.array_equal(a,b)
assert jnp.array_equal(a, 2.+3.*jax.random.normal(key_a,(20000,)))

The broad mean bound is five analytic standard errors for this independent normal sample; it is not a guarantee for arbitrary Monte Carlo samples.

Explain the APIs in this step
  • jax.random.split(key, num=...): Derives new keys for independent random operations; using the same key again repeats its random stream.
  • jax.random.key(seed): Creates a typed explicit PRNG key from a seed.
  • jax.random.normal(key, shape, dtype=...): Draws normal random values using an explicit PRNG key.
  • array.mean(axis=..., keepdims=...): Averages values along the selected axis; that axis is removed unless keepdims=True.

3 Compare an independent calculation

Append the independent scalar reference and stable mixture calculation, then run python main.py. Inspect the density, mean and variance.

main.py · append build step 3
# Step 3 — 3. Compare an independent calculation
actual = normal_logpdf(jnp.array([2.,5.]),2.,3.)
reference = [-math.log(3*math.sqrt(2*math.pi)), -.5-math.log(3*math.sqrt(2*math.pi))]
np.testing.assert_allclose(actual,reference,rtol=1e-6)
components = jnp.array([-1000.,-1001.])
stable = logsumexp(components)-jnp.log(2.)
assert jnp.isfinite(stable)
assert jnp.isneginf(jnp.log(jnp.exp(components).mean()))
print('sample mean, variance:',float(a.mean()),float(a.var()))
print('stable equal-mixture log density:',float(stable))

A log density near negative one thousand remains representable even when exponentiating it underflows in float32.

Explain the APIs in this step
  • jnp.array(values, dtype=...): Constructs a JAX array from Python/NumPy values.
  • np.testing.assert_allclose(actual, desired, rtol=..., atol=...): Checks a host-side numerical expectation and raises AssertionError when values differ beyond tolerance.
  • jnp.isfinite(x): Returns a JAX boolean array marking elements that are neither NaN nor infinite.
  • array.mean(axis=..., keepdims=...): Averages values along the selected axis; that axis is removed unless keepdims=True.
View the cumulative build in main.py

These blocks append in order to your separate practice file.

main.py · cumulative Python build
# Step 1 — 1. Define a density and its contract
import math
import numpy as np
import jax
import jax.numpy as jnp
from jax.scipy.special import logsumexp

def normal_logpdf(value, loc, scale):
    return -0.5*((value-loc)/scale)**2 - jnp.log(scale) - 0.5*jnp.log(2*jnp.pi)

# Step 2 — 2. Sample with explicit ownership
# Create or split explicit PRNG key(s) (`(key_a, key_b)`) for reproducible randomness.
key_a, key_b = jax.random.split(jax.random.key(23))
# Sample deterministic random values into `a` using an explicit PRNG key.
a = 2. + 3.*jax.random.normal(key_a, (20000,))
# Sample deterministic random values into `b` using an explicit PRNG key.
b = 2. + 3.*jax.random.normal(key_b, (20000,))
expected_mean, expected_variance = 2., 9.
assert abs(float(a.mean())-expected_mean) < 5*3/math.sqrt(a.size)
assert abs(float(a.var())-expected_variance) < .4
assert not jnp.array_equal(a,b)
assert jnp.array_equal(a, 2.+3.*jax.random.normal(key_a,(20000,)))

# Step 3 — 3. Compare an independent calculation
actual = normal_logpdf(jnp.array([2.,5.]),2.,3.)
reference = [-math.log(3*math.sqrt(2*math.pi)), -.5-math.log(3*math.sqrt(2*math.pi))]
np.testing.assert_allclose(actual,reference,rtol=1e-6)
components = jnp.array([-1000.,-1001.])
stable = logsumexp(components)-jnp.log(2.)
assert jnp.isfinite(stable)
assert jnp.isneginf(jnp.log(jnp.exp(components).mean()))
print('sample mean, variance:',float(a.mean()),float(a.var()))
print('stable equal-mixture log density:',float(stable))

Run the example

python3 phases/11-probability/01-distributions-and-sampling/code/main.py
About this command

This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.

Inspect the complete assembled example
main.py · complete Python example
# Step 1 — 1. Define a density and its contract
import math
import numpy as np
import jax
import jax.numpy as jnp
from jax.scipy.special import logsumexp

def normal_logpdf(value, loc, scale):
    return -0.5*((value-loc)/scale)**2 - jnp.log(scale) - 0.5*jnp.log(2*jnp.pi)

# Step 2 — 2. Sample with explicit ownership
# Create or split explicit PRNG key(s) (`(key_a, key_b)`) for reproducible randomness.
key_a, key_b = jax.random.split(jax.random.key(23))
# Sample deterministic random values into `a` using an explicit PRNG key.
a = 2. + 3.*jax.random.normal(key_a, (20000,))
# Sample deterministic random values into `b` using an explicit PRNG key.
b = 2. + 3.*jax.random.normal(key_b, (20000,))
expected_mean, expected_variance = 2., 9.
assert abs(float(a.mean())-expected_mean) < 5*3/math.sqrt(a.size)
assert abs(float(a.var())-expected_variance) < .4
assert not jnp.array_equal(a,b)
assert jnp.array_equal(a, 2.+3.*jax.random.normal(key_a,(20000,)))

# Step 3 — 3. Compare an independent calculation
actual = normal_logpdf(jnp.array([2.,5.]),2.,3.)
reference = [-math.log(3*math.sqrt(2*math.pi)), -.5-math.log(3*math.sqrt(2*math.pi))]
np.testing.assert_allclose(actual,reference,rtol=1e-6)
components = jnp.array([-1000.,-1001.])
stable = logsumexp(components)-jnp.log(2.)
assert jnp.isfinite(stable)
assert jnp.isneginf(jnp.log(jnp.exp(components).mean()))
print('sample mean, variance:',float(a.mean()),float(a.var()))
print('stable equal-mixture log density:',float(stable))

Expected result: The sample mean is close to 22, the variance is close to 99, and the stable mixture log-density is approximately −1000.38-1000.38.

Recorded output and runtime details

Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.

2026-10-27T17:13:56.914419+00:00 · Python 3.14.3 · JAX 0.9.2 · 1 CPU device

sample mean, variance: 2.0067408084869385 9.035282135009766
stable equal-mixture log density: -1000.3799438476562
sample mean, variance: 2.0067408084869385 9.035282135009766
stable equal-mixture log density: -1000.3799438476562
peak density: 3.9894230365753174
-5.0 -2.379885673522949
PASS: probability-01

Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.

Stage 3 of 5 · Controlled ExperimentsPredict before running & test boundary conditions

Density can exceed one

Density can exceed one · Python experiment
# Experiment — Density can exceed one
peak = jnp.exp(normal_logpdf(0.,0.,.1))
assert peak > 1
print("peak density:",float(peak))

Expected: The peak density is approximately 3.993.99.

A narrow continuous density can exceed one; its area, not its height at a point, sums to one.

Separate sum and mixture

Separate sum and mixture · Python experiment
# Experiment — Separate sum and mixture
values = jnp.array([-2.,-3.])
independent_log = values.sum()
mixture_log = logsumexp(values)-jnp.log(2.)
assert jnp.allclose(independent_log,-5.)
assert mixture_log > -3.
print(float(independent_log),float(mixture_log))

Expected: The independent joint log likelihood is −5-5; the mixture log density is approximately −2.38-2.38.

Independence multiplies probabilities, while alternative mixture components add weighted probabilities.

Stage 4 of 5 · Hands-On Exercise & CheckpointModify the code yourself, solve transfer tasks & test your understanding
Hands-On Exercise · Modify & Verify Run in the declared lesson environment

Make it yours

Where to put your code: Modify your own main.py, or use the Make it yours cell in your downloaded notebook. Run your attempt in the declared environment and verify your prediction before opening the reference solution.

Change the sensor to mean −1-1, standard deviation 0.50.5, and draw 3000030000 readings from a new key. Predict the variance and the standard error of the sample mean, then verify both moments.

Show a writing guide and starter scaffold

How to write this exercise code

Learner-only inputs, outputs, checks and hints

Inputs

  • A seed for a fresh observation key; mean and standard deviation of the sensor.

Outputs

  • One reading vector, its sample variance, and the standard error of its sample mean.

Invariants to check

  • The input key is not reused for another random draw.
  • The reported standard error uses the number of readings, not the variance alone.

Hints

  • Split a key before sampling.
  • For independent readings, standard error scales as standard deviation divided by the square root of the sample count.
  • Compare empirical moments with finite-sample tolerances rather than exact equality.
Your attempt · complete the TODOs
def compare_sample_moments(key, mean, scale, count):
    """Return readings, sample variance, and standard error of the mean."""
    # TODO: split the key, draw readings, and compute both summaries.
    raise NotImplementedError
Reveal the reference solution
Reference solution
# Exercise solution
# Create or split explicit PRNG key(s) (`changed`) for reproducible randomness.
changed = -1.+.5*jax.random.normal(jax.random.key(51),(30000,))
assert abs(float(changed.mean())+1.) < 5*.5/math.sqrt(30000)
assert abs(float(changed.var())-.25) < .015

Detect a missing normalization

Transfer / diagnosis

Compare scales 11 and 22 at their common mean. Show why a residual-only objective incorrectly ties them.

Show a writing guide and starter scaffold

How to write this practice solution

Learner-only inputs, outputs, checks and hints

Inputs

  • The shared mean and two normal scales, 1 and 2.

Outputs

  • The two log densities at the common mean and the change in the normalized density across scales.

Invariants to check

  • The residual-only term is identical at the shared mean, while the full log densities differ because scale changes the normalizer.

Hints

  • Evaluate both normalized log densities at the same zero-residual point.
  • Compare the full expression with the residual-only term.
Your attempt · complete the TODOs
import jax.numpy as jnp

def compare_normal_scales_at_common_mean(normal_logpdf):
    # TODO: Evaluate scales one and two at their common mean and expose the residual-only tie.
    raise NotImplementedError
Show a hint

The residual is zero for both; retain −log⁡σ-\log\sigma.

Reveal reference solution and reasoning
Detect a missing normalization · Python solution
# Detect a missing normalization (Transfer / diagnosis): Scale changes density even with identical zero residuals.
scores = normal_logpdf(0.,0.,jnp.array([1.,2.]))
assert jnp.allclose(scores[0]-scores[1],jnp.log(2.))

Scale changes density even with identical zero residuals.

Demonstrate key ownership

Transfer / diagnosis

Produce two separately owned standard-normal draws and show replay of the first. Explain why replay alone does not test independence.

Show a writing guide and starter scaffold

How to write this practice solution

Learner-only inputs, outputs, checks and hints

Inputs

  • An explicit JAX key, two child-work labels, and a changed seed.

Outputs

  • Two reproducible but independent random arrays.

Invariants to check

  • Reusing a key replays its random values; splitting creates independent child streams.

Hints

  • Split once for each random operation. Verify replay and changed-seed behavior separately.
Your attempt · complete the TODOs
def check_key_ownership(key, seed):
    """Return two draws and a replay/changed-seed comparison."""
    # TODO: allocate child keys and record the random-stream checks.
    raise NotImplementedError
Show a hint

Split before use; compare arrays with the same shape.

Reveal reference solution and reasoning
Demonstrate key ownership · Python solution
# Demonstrate key ownership (Transfer / diagnosis)
# Create or split explicit PRNG key(s) (`(left, right)`) for reproducible randomness.
left,right = jax.random.split(jax.random.key(17))
# Sample deterministic random values into `first` using an explicit PRNG key.
first = jax.random.normal(left,(32,))
# Sample deterministic random values into `second` using an explicit PRNG key.
second = jax.random.normal(right,(32,))
assert jnp.array_equal(first,jax.random.normal(left,(32,)))
assert not jnp.array_equal(first,second)

Different split keys avoid accidental replay; one differing pair does not prove every statistical property of a generator.

Check your understanding

Which statement explains why a large Monte Carlo sample can have a precise mean while individual readings remain variable?

Choose one answer

Diagnose the result

If the simulated variance is near three instead of nine, inspect whether you multiplied by sqrt33 when the stated scale was three. If two supposedly independent arrays match exactly, inspect key reuse before changing the distribution. If a mixture likelihood becomes negative infinity, compare the log-sum-exp result before widening dtype.

Stage 5 of 5 · Keep Your EvidenceRecord your verification receipt & carry takeaways forward

Carry forward

  • A density is not a point probability.
  • Moment checks need sample size, units and a justified tolerance.
  • Log-space arithmetic and explicit keys make likelihood experiments inspectable.

Keep your evidence

Download evidence template

Seeded arrays, independent density values, moment discrepancies and underflow diagnosis. Keep the environment, observed outputs and your explanation of the figure.

Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.

Saved in this browser. Export or back up your portfolio anytime in My learning.

Save work for this lesson

Reference execution: CPU validation only. This worked reference is validated on CPU. Accelerator compatibility requires a separate source review and target execution receipt. Use the execution matrix to review device selection and data ownership before adapting a separate target program. Execution matrix and target qualification.

Primary references