Phase 16 / Workload operations
Health, logs, and workload observability
Revision v0.18.1 · Updated since your last visit. Review the changes; your progress is saved.
Lesson · about 85 minutes · CPU practice
Build useful summaries from structured events emitted by a real worker. Define throughput with explicit numerator, denominator and synchronization boundary.
Phase study guide: preparation, milestones and project evidence
Before you start
Before this: Accelerator jobs and runtime lifecycle
Hardware: CPU
Setup help: Python, notebook, or Cloud TPU VM
Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions require separate checks. Hardware receipts apply only to their recorded source, environment, hardware, and cases; they do not qualify every lesson or TPU configuration. See the TPU performance and hardware evidence guide. The optional Cloud TPU specialization explains target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.
Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txtpy -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txtAfter activation, run the complete worked example from that same folder:
python phases/16-operations/02-health-logs-and-workload-observability/code/main.py.\.venv\Scripts\python.exe phases/16-operations/02-health-logs-and-workload-observability/code/main.pyFor an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.
Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.
For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:
python -m pip install jupyterlab
python -m jupyter lab.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter labSelect the downloaded operations-02.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.
The worked reference for this lesson is CPU-validated. Some references explicitly configure logical CPU devices; setting JAX_PLATFORMS does not adapt them to a TPU. Follow the execution matrix and reviewed multi-host example before attempting a target run.
For accelerator preparation, study TPU setup, observed runtime receipts, checkpointed jobs and precision and retained traces. Current receipts cover bounded v5e1 launch, resume, and precision checks; four single-device Pallas cases; and a four-device v5e collective fixture. Each receipt is limited to its recorded source, environment, cases, and hardware. It does not qualify all lessons or TPU configurations. Cloud provisioning is optional for CPU learning.
Check: The healthy run records processed examples. The failure occurs at step , with checkpoint step , so one completed update must be replayed.
Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.
What you will be able to do
- Build useful summaries from structured events emitted by a real worker.
- Define throughput with explicit numerator, denominator and synchronization boundary.
- Estimate uncheckpointed work from progress and checkpoint events.
The problem
A process is alive and prints messages, yet useful training may have stopped. What signals distinguish real progress from noise, and what could be lost if the process fails now? We will measure synchronized updates, count examples and compare progress with the latest committed checkpoint using events emitted by actual runs.
The idea
Logs tell us what a process reported; accepted checkpoints tell us what can be resumed. Monitoring should connect these signals without treating them as interchangeable. Their difference helps estimate the consequences of a failure.
Keep observed progress separate from recoverable progress
Monitoring is useful when its counters identify distinct facts. progress.step is work completed in memory; a checkpoint event is state replaced on disk. In the failed drill, events report steps 1 through 5, but the accepted checkpoint is step 4. That one-step gap is one completed update, or 8 examples at this run's batch size, which may need replay.
The figure plots both counters against completed update number: progress reaches 5 while the saved line ends at 4. A flat checkpoint line between its writes is expected; a flat progress line may be a stall, blocked logger, or failed collector. Check the process and another signal before deciding which. The next useful question is whether the mismatch is only expected checkpoint cadence or a missing progress event.
An event needs meaning and units
Each progress record carries a step, loss, example count, synchronized update seconds, and a hash of the resulting state. A checkpoint record names the step actually accepted on disk. A heartbeat would only say that a process responded; it must not add examples or advance the training counter. The monotonic timestamps support durations within this process, not clock comparisons across machines. When you inspect a trace, use event type and unit together rather than treating every increasing number as progress.
Time completed computation
JAX can return control before device work has finished. The worker calls block_until_ready before stopping the update timer, so update_s includes dispatch plus completion of that compiled minibatch. It excludes import, compilation, JSON logging, and checkpoint writes; warmup has its own event. A dispatch-only timer would make the update look faster than the completed work actually was. Compare timers only when their start and stop boundaries match.
Two throughputs answer different questions
In the healthy run, 12 updates process 96 examples. Dividing by summed synchronized update time estimates the compiled update path; dividing the same 96 examples by supervisor wall time includes startup and checkpointing, so it is lower for this short job. The execution record reports that split directly. Neither rate is accelerator utilization: to answer a utilization question you need a device-specific counter, not examples divided by seconds.
Freshness is about committed progress
Checkpoint cadence is every two updates. The injected failure occurs after update 5, leaving step 4 as the newest committed state and one completed update, 8 examples, outside it. That gap estimates replayable work, not recovery time: a slow checkpoint write or a different batch duration changes seconds at risk. For a time estimate, use measured update durations and say whether the final in-flight update completed.
A declining loss is not a health guarantee
This fixture trains a line with sampled minibatches and momentum, so loss can rise when the next minibatch is harder or momentum overshoots. A lower loss does not verify fresh data or a durable checkpoint; a noisy loss does not prove a stall. Use the loss to inspect optimization, and event cadence plus the process handle to inspect liveness. The signals answer different questions.
Scope of this collector
This worker captures stdout and stderr in memory and parses a few dozen events after exit. That makes the record easy to inspect, but it does not provide streaming alerts or a durable fleet log. A long-lived service needs bounded streaming, a persistent sink, aligned timestamps, and monitoring for queue backpressure. The lesson's concrete result is the local event trace and its checkpoint gap, not a production telemetry service.
Observed progress and committed progress diverge before failure
Predict before looking: Will the checkpoint line always equal the progress line?
Read the figure
The x-axis is completed update number, and the y-axis is step index. The failed trace has progress at steps 1–5, while the checkpoint staircase advances at 2 and 4 and ends at 4. At failure, the vertical gap is one update, or 8 examples for this batch size. These are step counts from the emitted events, not estimated utilization.
Connect it to the computation
A checkpoint plateau between writes is expected; the final gap matters because failure makes update 5 replayable from step 4. The run record reports the incident as failed with 40 examples observed, 8 uncheckpointed, and a completed-checkpoint boundary at step 4. If cadence changed from every two updates to every one, predict that the final gap would shrink to zero, at the cost of more checkpoint writes. The plot does not measure replay time or storage durability.
Figure data and execution details
Open full-size figureDownload data and provenance
The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.
Your build: Create a separate main.py in your practice folder. Append each block in order and run python main.py in the lesson environment. The downloaded script is the complete reference, so keep it separate from your attempt.
1 Compute a rate over one declared interval
Use one numerator and the matching measured interval.
def rate_per_second(count, seconds):
return count / seconds if seconds else 0.The helper is used for both synchronized update time and whole-job wall time.
2 Compute the recoverability gap
Compare observed progress with the accepted checkpoint.
def checkpoint_gap(last_step, checkpoint_step):
return max(0, last_step - checkpoint_step)A log line and a durable checkpoint represent different boundaries.
3 Translate missing updates into examples
Change the batch size only after interpreting the logged steps.
def checkpointed_examples(last_step, checkpoint_step, batch_size):
return checkpoint_gap(last_step, checkpoint_step) * batch_sizeThe count is meaningful because the fixture uses a fixed batch size.
View the cumulative build in main.py
These blocks append in order to your separate practice file.
def rate_per_second(count, seconds):
return count / seconds if seconds else 0.
def checkpoint_gap(last_step, checkpoint_step):
return max(0, last_step - checkpoint_step)
def checkpointed_examples(last_step, checkpoint_step, batch_size):
return checkpoint_gap(last_step, checkpoint_step) * batch_sizeRun the example
python3 phases/16-operations/02-health-logs-and-workload-observability/code/main.pyAbout this command
This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.
Inspect the complete assembled example
# Complete runnable example (operations-02)
"""Bounded local JAX subprocess supervisor and lifecycle verifier."""
from pathlib import Path
import hashlib
import json
import math
import os
import subprocess
import sys
import tempfile
import time
def digest(value):
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode()).hexdigest()
def atomic_json(path, value):
"""Replace one local JSON file only after its bytes have been flushed."""
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
fd, temporary = tempfile.mkstemp(prefix=path.name+'.', suffix='.tmp', dir=path.parent)
try:
with os.fdopen(fd, 'w') as stream:
json.dump(value, stream, sort_keys=True, allow_nan=False)
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary, path)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
def rate_per_second(count, seconds):
return count / seconds if seconds else 0.
def checkpoint_gap(last_step, checkpoint_step):
return max(0, last_step - checkpoint_step)
def checkpointed_examples(last_step, checkpoint_step, batch_size):
return checkpoint_gap(last_step, checkpoint_step) * batch_size
# This file is materialized only inside the caller-owned temporary run folder.
WORKER = r'''
import hashlib, json, os, sys, time
from pathlib import Path
import jax
import jax.numpy as jnp
import numpy as np
root = Path(sys.argv[1])
cfg = json.loads((root/'config.json').read_text())
worker_hash = hashlib.sha256(Path(__file__).read_bytes()).hexdigest()
def emit(event, **fields):
print(json.dumps(dict(event=event, monotonic_s=time.monotonic(), **fields)), flush=True)
def digest(value):
return hashlib.sha256(json.dumps(value,sort_keys=True,separators=(',', ':'),allow_nan=False).encode()).hexdigest()
def write_checkpoint(state):
target = root/'checkpoint.json'
temporary = root/'checkpoint.pending'
with temporary.open('w') as stream:
json.dump(dict(state,checksum=digest(state)),stream,sort_keys=True,allow_nan=False)
stream.flush()
os.fsync(stream.fileno())
os.replace(temporary,target)
emit('checkpoint',step=state['step'],state_hash=digest(state))
try:
emit('started',pid=os.getpid(),backend=jax.default_backend(),device_count=jax.device_count())
if jax.default_backend() != cfg['backend'] or jax.device_count() < cfg['min_devices']:
raise ValueError('runtime backend/device contract failed')
x = jnp.linspace(-1.,1.,64)
y = 2*x+1
data_hash = digest(dict(x=np.asarray(x).tolist(),y=np.asarray(y).tolist()))
training = {name:cfg[name] for name in ('seed','learning_rate','momentum','batch_size')}
config_hash = digest(training)
if cfg['resume']:
saved = json.loads((root/'checkpoint.json').read_text())
checksum = saved.pop('checksum',None)
if checksum != digest(saved) or saved.get('worker_hash') != worker_hash:
raise ValueError('checkpoint checksum/source mismatch')
if saved['schema_version'] != 1 or saved['config_hash'] != config_hash or saved['data_hash'] != data_hash:
raise ValueError('checkpoint provenance mismatch')
state = saved
step = saved['step']
params=jnp.array(saved['params'])
velocity=jnp.array(saved['velocity'])
key=jnp.array(saved['key'],dtype=jnp.uint32)
if not isinstance(step,int) or isinstance(step,bool) or step < 0 or step > cfg['steps'] or params.shape != (2,) or velocity.shape != (2,) or key.shape != (2,):
raise ValueError('invalid checkpoint state shape or step')
if not bool(jnp.all(jnp.isfinite(params))) or not bool(jnp.all(jnp.isfinite(velocity))):
raise ValueError('nonfinite checkpoint state')
emit('restored',step=step,state_hash=digest(saved))
else:
step=0
params=jnp.zeros(2)
velocity=jnp.zeros(2)
key=jax.random.PRNGKey(cfg['seed'])
def update(params,velocity,key):
key, sample = jax.random.split(key)
indices=jax.random.choice(sample,64,(cfg['batch_size'],),replace=False)
objective=lambda p:jnp.mean((p[0]*x[indices]+p[1]-y[indices])**2)
loss, grad=jax.value_and_grad(objective)(params)
velocity=cfg['momentum']*velocity+grad
params=params-cfg['learning_rate']*velocity
return params,velocity,key,loss
compiled=jax.jit(update)
# Compile and synchronize once without consuming actual training state.
begin=time.perf_counter()
warm=compiled(params,velocity,key)
jax.block_until_ready(warm)
emit('ready',compile_warmup_s=time.perf_counter()-begin,step=step)
while step < cfg['steps']:
if cfg.get('stall_at') == step:
emit('stall_injected',step=step)
time.sleep(cfg.get('stall_seconds',10.))
begin=time.perf_counter()
params,velocity,key,loss=compiled(params,velocity,key)
jax.block_until_ready((params,velocity,key,loss))
elapsed=time.perf_counter()-begin
step+=1
state=dict(schema_version=1,worker_hash=worker_hash,step=step,params=np.asarray(params).tolist(),velocity=np.asarray(velocity).tolist(),key=np.asarray(key).tolist(),config_hash=config_hash,data_hash=data_hash)
emit('progress',step=step,loss=float(loss),examples=cfg['batch_size'],update_s=elapsed,state_hash=digest(state))
if step % cfg['checkpoint_every'] == 0 or step == cfg['steps']:
write_checkpoint(state)
if cfg.get('fail_after') == step:
raise RuntimeError('controlled failure after update')
write_checkpoint(state)
emit('completed',step=step,state_hash=digest(state))
except Exception as error:
emit('failed',kind=type(error).__name__,message=str(error))
raise SystemExit(23)
'''
# Step 3 — Add bounded launch and result collection
def default_config(**changes):
cfg=dict(seed=3,learning_rate=.04,momentum=.8,batch_size=8,steps=8,
checkpoint_every=2,backend='cpu',min_devices=1,resume=False,
fail_after=None,stall_at=None,stall_seconds=10.)
# Apply configuration overrides before validating the resulting fields.
cfg.update(changes)
for name in ('steps','checkpoint_every','batch_size','min_devices'):
if not isinstance(cfg[name],int) or isinstance(cfg[name],bool) or cfg[name] < 1:
raise ValueError(name+' must be a positive integer')
if cfg['batch_size'] > 64 or not 0 <= cfg['momentum'] < 1 or not 0 < cfg['learning_rate'] < 1:
raise ValueError('invalid training configuration')
return cfg
def launch(root, config=None, timeout=10.):
root=Path(root)
root.mkdir(parents=True,exist_ok=True)
cfg=default_config(**(config or {}))
if timeout <= 0:
raise ValueError('timeout must be positive')
atomic_json(root/'config.json',cfg)
worker=root/'worker.py'
worker.write_text(WORKER)
env=dict(os.environ,JAX_PLATFORMS=cfg['backend'],PYTHONUNBUFFERED='1')
started=time.perf_counter()
process=subprocess.Popen([sys.executable,str(worker),str(root)],stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True,env=env)
timed_out=False
try:
stdout,stderr=process.communicate(timeout=timeout)
except subprocess.TimeoutExpired:
timed_out=True
process.terminate()
try:
stdout,stderr=process.communicate(timeout=1.)
except subprocess.TimeoutExpired:
process.kill()
stdout,stderr=process.communicate(timeout=1.)
wall=time.perf_counter()-started
events=[]
malformed=[]
for line in stdout.splitlines():
if not line.strip():
continue
try:
event=json.loads(line)
if not isinstance(event,dict) or 'event' not in event:
raise ValueError('not an event')
events.append(event)
except (ValueError,TypeError):
malformed.append(line)
status='timed_out' if timed_out else 'completed' if process.returncode == 0 and not malformed and events and events[-1]['event']=='completed' else 'failed'
result=dict(status=status,returncode=process.returncode,wall_s=wall,events=events,stderr=stderr,
worker_hash=hashlib.sha256(WORKER.encode()).hexdigest(),config=cfg,malformed_stdout=malformed)
atomic_json(root/'run.json',result)
return result
# Step 4 — Summarize event boundaries
def summarize(result):
updates=[e for e in result['events'] if e['event']=='progress']
checkpoints=[e['step'] for e in result['events'] if e['event']=='checkpoint']
count=sum(e['examples'] for e in updates)
update_s=sum(e['update_s'] for e in updates)
latest=max((e['step'] for e in updates),default=0)
checkpoint=max(checkpoints,default=0)
return dict(status=result['status'],updates=len(updates),examples=count,last_step=latest,checkpoint_step=checkpoint,
uncheckpointed_updates=checkpoint_gap(latest,checkpoint),
uncheckpointed_examples=checkpointed_examples(latest,checkpoint,result['config']['batch_size']),update_s=update_s,
update_examples_per_s=rate_per_second(count,update_s),
job_examples_per_s=rate_per_second(count,result['wall_s']),
update_duty_fraction=update_s/result['wall_s'] if result['wall_s'] else 0.)
# Step 5 — Run a healthy job and a controlled incident
# Keep this run’s database and artifacts in a scratch directory removed on exit.
with tempfile.TemporaryDirectory(prefix='ops-signals-') as folder:
observed=launch(Path(folder)/'healthy',dict(steps=12))
interrupted=launch(Path(folder)/'failed',dict(steps=12,fail_after=5))
healthy=summarize(observed)
incident=summarize(interrupted)
progress=[e for e in observed['events'] if e['event']=='progress']
assert healthy['examples']==96 and healthy['updates']==12
assert incident['last_step']==5 and incident['checkpoint_step']==4
assert incident['uncheckpointed_updates']==1
assert healthy['job_examples_per_s'] < healthy['update_examples_per_s']
print('healthy:',healthy)
print('incident:',incident)Expected result: The healthy run records processed examples. The failure occurs at step , with checkpoint step , so one completed update must be replayed.
Recorded output and runtime details
Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.
healthy: {'status': 'completed', 'updates': 12, 'examples': 96, 'last_step': 12, 'checkpoint_step': 12, 'uncheckpointed_updates': 0, 'uncheckpointed_examples': 0, 'update_s': 0.020353708881884813, 'update_examples_per_s': 4716.585097934746, 'job_examples_per_s': 86.91440289879009, 'update_duty_fraction': 0.018427400565050198}
incident: {'status': 'failed', 'updates': 5, 'examples': 40, 'last_step': 5, 'checkpoint_step': 4, 'uncheckpointed_updates': 1, 'uncheckpointed_examples': 8, 'update_s': 0.0008766651153564453, 'update_examples_per_s': 45627.457166168075, 'job_examples_per_s': 38.270822211904765, 'update_duty_fraction': 0.0008387673692296375}
examples / synchronized seconds: 96 0.020353708881884813
measured update duty fraction, not hardware utilization: 0.018427400565050198
PASS: operations-02
Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.
Recompute throughput independently
# Experiment — Recompute throughput independently
total_examples=sum(e['examples'] for e in progress)
total_time=sum(e['update_s'] for e in progress)
assert abs(total_examples/total_time-healthy['update_examples_per_s']) < 1e-6
assert all(e['update_s']>0 for e in progress)
print('examples / synchronized seconds:',total_examples,total_time)Expected: The summary matches the independently accumulated numerator and denominator.
Summing work and time weights slow steps appropriately; an unweighted mean of rates generally answers a different question.
Locate the incident boundary
# Experiment — Locate the incident boundary: A progress log is not a committed state artifact.
names=[e['event'] for e in interrupted['events']]
assert names[-1]=='failed'
assert [e['step'] for e in interrupted['events'] if e['event']=='checkpoint']==[2,4]
assert [e['step'] for e in interrupted['events'] if e['event']=='progress'][-1]==5Expected: The event order exposes exactly one completed update after the latest checkpoint.
A progress log is not a committed state artifact. Recovery must select the committed boundary.
Make it yours
Where to put your code: Modify your own main.py, or use the Make it yours cell in your downloaded notebook. Run your attempt in the declared environment and verify your prediction before opening the reference solution.
Use the emitted healthy and incident logs to recompute synchronized examples/s and the step-5 versus checkpoint-4 gap. Change checkpoint_every from 2 to 3, rerun, and describe how the actual failure log changes the recovery gap.
Show a writing guide and starter scaffold
How to write this exercise code
Learner-only inputs, outputs, checks and hints
Inputs
- The synchronized-update duration and supervisor wall-clock duration for the same run.
Outputs
- The fraction of supervisor wall time spent in synchronized updates and a precise list of excluded intervals.
Invariants to check
- The duty fraction is synchronized-update time divided by supervisor wall time, and does not claim to include time outside that boundary.
Hints
- Use durations measured over the same run window.
- Name setup, queue, and checkpoint intervals only if the supervisor timing boundary excludes them.
def compute_update_duty_fraction(synchronized_update_seconds, supervisor_wall_seconds):
# TODO: Divide synchronized update time by supervisor wall time and state what that interval excludes.
raise NotImplementedErrorReveal the reference solution
# Exercise solution
duty=sum(e['update_s'] for e in progress)/observed['wall_s']
assert 0 < duty < 1
assert abs(duty-healthy['update_duty_fraction']) < 1e-9
print('measured update duty fraction, not hardware utilization:',duty)Catch an average-of-rates error
Transfer / diagnosis
Two updates process examples each in and seconds. Compare the average rate with the rate of all completed work.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- Two updates process examples each in and seconds. Compare the average rate with the rate of all completed work.
Outputs
- Unweighted mean of rates and total-work throughput.
Invariants to check
- Aggregate throughput is total examples divided by total elapsed seconds.
Hints
- Sum examples and seconds first.
- Sum numerators and denominators before dividing.
import math
def practice_catch_an_average_of_rates_error(example_counts, elapsed_seconds):
"""Complete the catch an average-of-rates error check."""
# TODO: ('Compute mean of the two update rates and total examples divided by total seconds for the 8-example updates.', 'Both rate summaries with the total-work rate identified.', 'The aggregate rate uses summed examples over summed elapsed time.')
raise NotImplementedErrorShow a hint
Sum examples and seconds first.
Reveal reference solution and reasoning
# Catch an average-of-rates error (Transfer / diagnosis)
durations=[1.,3.]
counts=[8,8]
wrong=sum(n/t for n,t in zip(counts,durations))/2
right=sum(counts)/sum(durations)
assert abs(wrong-16/3)<1e-12 and right==4.The short step gets excessive influence when rates are averaged without accounting for their duration.
Change the checkpoint cadence
Transfer / diagnosis
Run with checkpoint cadence and fail after update . Predict the committed step and replay count.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- Run with checkpoint cadence and fail after update . Predict the committed step and replay count.
Outputs
- Committed checkpoint step and amount of replay after failure.
Invariants to check
- A failure after update five cannot advance beyond the last completed cadence checkpoint.
Hints
- Expect the most recent multiple of the checkpoint interval before the failure.
- Distinguish attempted updates from committed state.
from pathlib import Path
import hashlib
import json
import math
import os
import time
import numpy as np
def practice_change_the_checkpoint_cadence(updates, checkpoint_every, failure_after):
"""Complete the change the checkpoint cadence check."""
# TODO: ('Simulate checkpoint interval three and failure after update five, then replay from the last committed step.', 'Committed step and replay count from the declared cadence.', 'Only committed checkpoints survive the injected failure.')
raise NotImplementedErrorShow a hint
Expect the most recent multiple of the checkpoint interval before the failure.
Reveal reference solution and reasoning
# Change the checkpoint cadence (Transfer / diagnosis)
# Keep this run’s database and artifacts in a scratch directory removed on exit.
with tempfile.TemporaryDirectory() as folder:
changed=launch(folder,dict(checkpoint_every=3,fail_after=5))
stats=summarize(changed)
assert stats['checkpoint_step']==3 and stats['uncheckpointed_updates']==2The run tests a changed operational policy rather than assuming every failure loses exactly one update.
Diagnose the result
If throughput is implausibly high, check synchronization and whether examples were counted twice. If checkpoint freshness looks perfect during a failure, verify that checkpoint events are emitted only after replacement completes. If timestamps appear out of order across hosts, do not subtract unrelated monotonic clocks.
Carry forward
- Choose metrics that answer a concrete operational decision.
- Specify the timing boundary before reporting a throughput or utilization-like ratio.
Keep your evidence
Keep raw structured events, observed work counts, synchronized timing units and checkpoint lag. Derive job and update throughput with different boundaries.
Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.
Saved in this browser. Export or back up your portfolio anytime in My learning.
Reference execution: CPU validation only. This worked reference is validated on CPU. Accelerator compatibility requires a separate source review and target execution receipt. Use the execution matrix to review device selection and data ownership before adapting a separate target program. Execution matrix and target qualification.