Check your starting point

The weights restore exactly, but the next loss differs. Which state should you inspect before blaming numerical nondeterminism?

Compare your reasoning

Compare optimizer moments and count, random keys, data permutation/cursor, preprocessing and configuration. The same weights with a different next batch are a different transition.

If this is unfamiliar, work through Save model, optimizer, and random state before attempting the project.

Build in stages

1. Preserve record identity and position

Inspect record IDs through shuffling and batching. State the final-batch policy and save iterator state only at the intended consumption boundary.

2. Prove a committed continuation

Save complete state, restore in a fresh process and compare future IDs and updates across an epoch boundary. Reject an incomplete or incompatible checkpoint before publishing it.

3. Connect continuation to lineage

Retrieve an MLflow run and its artifact, compare data/source identities and metric-step conventions, and distinguish a registry selection from active deployment.

Track experiments and lineage with MLflow

Try a changed condition

Make a prediction before opening the reasoning. Keep your answer and the evidence you would collect.

The checkpoint is valid, but the dataset was reordered without changing its filename. Is the saved cursor sufficient?

Compare an approach

No. The cursor is meaningful relative to a dataset identity and ordering configuration. Verify content/version identity first; reject mismatches rather than silently resuming on different records.

Diagnose a likely failure

Symptom: The advertised checkpoint step is newer than the restored state.

Check next: Inspect completion and publication order. An asynchronous save request is not proof that the artifact is durable or restorable.

Decide what is ready

Use sharded-training recovery stages and the MLflow extension. Keep uninterrupted and restarted trajectories, a corrupt/incompatible rejection, and a retrieved artifact tied to its run.

Where to put your project code

Write your code in: projects/recovery-audit/starter/model.py (copy projects/recovery-audit/starter/model.py first: cp projects/recovery-audit/starter/model.py projects/recovery-audit/starter/model.py).

Verify your implementation: python3 projects/recovery-audit/tests/check.py --stage all --impl projects/recovery-audit/starter/model.py

Run the staged project checks on your implementation in projects/recovery-audit/starter/model.py and record your transfer experiment results in your portfolio.

Open Resume optimizer and PRNG state across an interrupted training update

Open Ship a tracked and containerized model release

Open Resume and profile a sharded training run

Open Reproducible JAX/Flax experiments workshop