Deployment
Deployment Guide
Remote Execution Backends
flaxchat supports local execution plus two CLI-driven accelerator paths:
graph TD
USER[Your Laptop] --> |"pixi run"| LOCAL[Local CPU/GPU]
USER --> |"kaggle CLI"| KAGGLE[Kaggle TPU v5e-8]
USER --> |"tpuz CLI/API"| GCP[GCP TPU Pod]
All backends emit the same serializable flaxchat.launch.LaunchSpec; adapters
handle provisioning and transport while training stays in shared services.
1. Local (laptop/workstation)
pixi install
python -m scripts.run_tinystories --layers=4 --pretrain-steps=1000
2. Kaggle training and accelerator validation
python -m pip install kaggle
python -m scripts.train_kaggle \
--kernel-id OWNER/flaxchat-training --accelerator tpu --steps 100 --no-wait
# Resume monitoring later without submitting another paid/quota-consuming run.
python -m scripts.kaggle_tpu_tests \
--kernel-id OWNER/flaxchat-training --resume-monitor
# Full acceptance bundle (run only for release candidates or accelerator changes).
python -m scripts.kaggle_tpu_tests \
--kernel-id OWNER/flaxchat-tpu-tests --accelerator tpu --wait
Both generated private kernels check out an exact 40-character Git revision.
The acceptance kernel installs
TPU JAX, verifies eight devices, runs the entire suite once, executes the pinned
TinyStories end-to-end pipeline, and records attention/speculative benchmarks.
Logs, artifacts, JUnit XML, and a JSON summary are downloaded together.
Transient Kaggle API outages are persisted locally and retried; --resume-monitor
reconnects without creating a duplicate kernel version.
Web chat service
scripts.chat_web is an application factory: importing it does not parse
arguments or load model state. Production adapters should inject a
manifest-verified ChatService and set bounded input, generation, concurrency,
and output-buffer limits through WebSettings.
The WebSocket protocol emits token, done, or structured error events.
Error codes are invalid_json, invalid_request, context_overflow,
overloaded, request_error, and model_error; internal model exceptions are
not exposed. Disconnects signal cooperative cancellation into cached decoding.
3. GCP TPU Pod
The production flex-start launcher, pinned environment, worker coordination,
artifact collection, and teardown guardrails live in infra/tpu/.
Python API
from tpuz import TPU
tpu = TPU("flaxchat-d24", accelerator="v6e-8")
tpu.up()
tpu.setup(extra_pip="flaxchat")
tpu.run("python -m scripts.pretrain --depth=24", sync=".")
TPU Types
| Accelerator | Chips | Workers | Free Tier | Zones |
|---|---|---|---|---|
v4-8 |
4 | 1 | On-demand | us-central2-b |
v4-32 |
16 | 4 | No | us-central2-b |
v5litepod-8 |
8 | 1 | TRC | us-central1-a |
v5litepod-64 |
64 | 8 | TRC | us-central1-a |
v6e-8 |
8 | 1 | TRC | europe-west4-a |
v6e-64 |
64 | 8 | No | europe-west4-a |
Multi-Host Training
For pods with multiple processes, JAX distributed initialization coordinates
hosts and FlaxChat builds one global SPMD mesh. Each host consumes alternating
parquet row groups and reconstructs global batches with
jax.make_array_from_process_local_data.
Preemption Recovery
graph TD
RUNNING[Training Running] --> |"Poll every 60s"| CHECK{VM State?}
CHECK --> |READY| ALIVE{Process alive?}
ALIVE --> |Yes| RUNNING
ALIVE --> |No, completed| DONE[Done]
ALIVE --> |No, crashed| RESTART[Restart training]
CHECK --> |PREEMPTED| RECOVER[Delete → Recreate → Setup → Resume]
RECOVER --> RUNNING
RESTART --> RUNNING
Relies on Orbax checkpoints for resume — training picks up from last saved step.
Export
Checkpoint Formats
| Format | File | Use Case |
|---|---|---|
| JAX checkpoint | model.pkl |
Resume training, load in flaxchat |
| NumPy weights | weights.npz |
Portable, load anywhere |
| StableHLO | model.stablehlo |
LiteRT/TFLite conversion input |
LiteRT/TFLite Conversion
# On Linux with TensorFlow:
python -m scripts.convert_to_tflite \
--checkpoint=exports/model.pkl \
--output=exports/model.tflite