Check your starting point
Step 1 of your TPU job takes 14 seconds while steps 2–50 take 11 milliseconds each. How should your launcher record timing?
Compare your reasoning
Run and synchronize one warmup call before the timed loop and log it as a separate warmup event so XLA compilation does not pollute steady-state step latency.
If this is unfamiliar, work through Stage and launch checkpointed TPU jobs before attempting the project.
Build in stages
1. Launch resumable TPU jobs with warmup separation
Run run_tpu_ready_job, log warmup separately from timed steps, and verify that a checkpoint-resumed run matches an uninterrupted control run bit-for-bit.
2. Stage course scripts and launch multi-host Pod slices
Classify distributed-02 as CPU-only and leave unreviewed scripts blocked. Generate a TPU command only for a source-reviewed target with an explicit backend and pinned dependencies; distinguish --worker=all topology discovery from a collective training receipt with per-process data ownership and checked global updates.
Try a changed condition
Make a prediction before opening the reasoning. Keep your answer and the evidence you would collect.
A four-process job saves global step 20 but only three workers report it. Can it become the accepted recovery checkpoint?
Compare an approach
A step label does not establish a globally committed snapshot. Require all participating workers, a completed compatible checkpoint and the declared restore protocol. Keep the previous accepted checkpoint on failure. The existing multi-host receipt tests a bounded same-process restore; fresh-process worker-loss recovery remains a separate qualification.
Diagnose a likely failure
Symptom: A resumed run loads the saved model weights w, but its next loss diverges from the uninterrupted control run.
Check next: Verify that the checkpoint also restored optimizer momentum v and step counter step so the deterministic batch schedule continues at the exact next step.
Decide what is ready
Keep the CPU checkpoint-resume comparison from tpu-02, the per-script execution matrix and rejected unsafe routes from tpu-05, and any target receipt only when its source hash, dependencies, backend, topology, and input ownership match the qualified program. Multi-host qualification remains separate.
Where to put your project code
Write your code in: projects/tpu-workflow-audit/starter/model.py (copy projects/tpu-workflow-audit/starter/model.py first: cp projects/tpu-workflow-audit/starter/model.py projects/tpu-workflow-audit/starter/model.py).
Verify your implementation: python3 projects/tpu-workflow-audit/tests/check.py --stage all --impl projects/tpu-workflow-audit/starter/model.py
Run the staged project checks on your implementation in projects/tpu-workflow-audit/starter/model.py and record your transfer experiment results in your portfolio.
Open Operate and recover a local JAX workload
What still needs further work
Continue to Phase 21 (TPU Systems: Generations, Memory, Precision & XProf) to budget HBM, test BF16/INT8 precision, and capture XProf traces.