Course · TPU Workflows: Launching Jobs & Checkpointed ExperimentsPhase 20 · Study guide

Can you launch a JAX job over SSH, separate XLA warmup from steady-state timing, and resume cleanly after interruption?

Cloud TPU Mini-Course · Step 2 of 3Cloud TPU VM + CPU verifierBite-sized · 2 lessons (~45m)

Revision v0.18.1 ·

A cloud training job needs a repeatable path from local source to running process to retained result. Follow which files and dependencies reach each worker, when warmup ends, which state is checkpointed and where the resulting evidence is copied before teardown. The single-host exercise establishes a bounded continuation. Multi-host work additionally requires agreement across participants about topology, global steps and accepted checkpoint boundaries. A topology-discovery command and a collective training receipt answer different questions; use the observed state to decide what the next experiment can safely assume.

Preparation: TPU Setup: Provisioning & Runtime Verification

Runtime: CPU workflow planner; target execution only for source-reviewed programs with matching backend, dependencies, topology, and data-ownership contracts

Check your starting point

Step 1 of your TPU job takes 14 seconds while steps 2–50 take 11 milliseconds each. How should your launcher record timing?

Compare your reasoning

Run and synchronize one warmup call before the timed loop and log it as a separate warmup event so XLA compilation does not pollute steady-state step latency.

If this is unfamiliar, work through Stage and launch checkpointed TPU jobs before attempting the project.

Build in stages

1. Launch resumable TPU jobs with warmup separation

Run run_tpu_ready_job, log warmup separately from timed steps, and verify that a checkpoint-resumed run matches an uninterrupted control run bit-for-bit.

2. Stage course scripts and launch multi-host Pod slices

Classify distributed-02 as CPU-only and leave unreviewed scripts blocked. Generate a TPU command only for a source-reviewed target with an explicit backend and pinned dependencies; distinguish --worker=all topology discovery from a collective training receipt with per-process data ownership and checked global updates.

Try a changed condition

Make a prediction before opening the reasoning. Keep your answer and the evidence you would collect.

A four-process job saves global step 20 but only three workers report it. Can it become the accepted recovery checkpoint?

Compare an approach

A step label does not establish a globally committed snapshot. Require all participating workers, a completed compatible checkpoint and the declared restore protocol. Keep the previous accepted checkpoint on failure. The existing multi-host receipt tests a bounded same-process restore; fresh-process worker-loss recovery remains a separate qualification.

Diagnose a likely failure

Symptom: A resumed run loads the saved model weights w, but its next loss diverges from the uninterrupted control run.

Check next: Verify that the checkpoint also restored optimizer momentum v and step counter step so the deterministic batch schedule continues at the exact next step.

Decide what is ready

Keep the CPU checkpoint-resume comparison from tpu-02, the per-script execution matrix and rejected unsafe routes from tpu-05, and any target receipt only when its source hash, dependencies, backend, topology, and input ownership match the qualified program. Multi-host qualification remains separate.

Where to put your project code

Write your code in: projects/tpu-workflow-audit/starter/model.py (copy projects/tpu-workflow-audit/starter/model.py first: cp projects/tpu-workflow-audit/starter/model.py projects/tpu-workflow-audit/starter/model.py).

Verify your implementation: python3 projects/tpu-workflow-audit/tests/check.py --stage all --impl projects/tpu-workflow-audit/starter/model.py

Run the staged project checks on your implementation in projects/tpu-workflow-audit/starter/model.py and record your transfer experiment results in your portfolio.

Open Gate simulated multi-worker checkpoint publication on complete receipts

Open Operate and recover a local JAX workload

What still needs further work

Continue to Phase 21 (TPU Systems: Generations, Memory, Precision & XProf) to budget HBM, test BF16/INT8 precision, and capture XProf traces.