flaxchat
flaxchat
A minimal, end-to-end LLM training harness for Google Cloud TPU pods, built on JAX/Flax NNX.
Port of nanochat to the JAX ecosystem with full feature parity plus speculative decoding.
Part of the 2026 Q1 TPU Research Sprint, supported by the Google AI Developer Programs team.
Quick Start
pixi install
pixi run test
pixi run test-e2e
pixi run -- python -m scripts.run_tinystories --layers=8 --pretrain-steps=5000
Core Features
GPT Architecture (all nanochat features)
- Rotary Embeddings (RoPE), Group-Query Attention (GQA), QK Normalization
- ReLU^2 MLP, Value Embeddings (ResFormer), Sliding Window (SSSL pattern)
- Per-layer Residual Scaling, Smear, Backout, Logit Soft-capping
- Gradient Checkpointing (
nnx.remat,dots_saveable)
Optimizer
- Mixed Muon + AdamW via
optax.multi_transform - Muon: Polar Express orthogonalization + NorMuon variance reduction
- Per-group LR, betas, weight decay. Warmup -> constant -> warmdown schedule
Inference Engine (4 modes)
| Mode | Function | Speed | Description |
|---|---|---|---|
| Padded | generate() |
Measured per run | Simple, for debugging |
| KV-cached | generate_with_cache() |
Measured per run | Python loop with KV cache |
| Fully JIT | generate_fast() |
Measured per run | jax.lax.while_loop, no Python overhead |
| Speculative | generate_speculative() |
Benchmark per pairing | Batched main-model verification |
Tool Use (streaming)
engine = Engine(model, tokenizer)
for tokens, masks in engine.generate(prompt_ids, num_samples=3):
# Automatic calculator; generated Python is disabled by default
pass
Guarded Code Execution
from flaxchat.execution import execute_code
result = execute_code("print(sum(range(10)))", timeout=5.0, trusted=True)
# ExecutionResult(success=True, stdout="45\n")
This is a best-effort reliability guard for reviewed code, not a security sandbox. Untrusted execution is disabled unless an external isolation backend is explicitly configured.
Parallelism (default, not optional)
mesh = compute_init() # auto mesh over ALL devices
# Data parallel, FSDP, multi-host — all automatic via JAX SPMD
Depth-Based Config
config = FlaxChatConfig.from_depth(depth=12)
# -> 12 layers, 768 dims, 6 heads, ~79M params
Pipeline
| Stage | Script | Description |
|---|---|---|
| Tokenizer | scripts/tok_train.py |
Train BPE tokenizer |
| Pretrain | scripts/pretrain.py |
Pretrain on ClimbMix-400B / TinyStories |
| SFT | scripts/sft.py |
Supervised fine-tuning |
| RL | scripts/rl.py |
GRPO/REINFORCE on GSM8K |
| Eval | scripts/eval.py |
MMLU, ARC, GSM8K, HumanEval, CORE |
| Chat | scripts/chat_web.py |
FastAPI WebSocket UI |
| Export | scripts/convert_to_tflite.py |
LiteRT/TFLite for edge |
Evaluation Tasks
| Task | Type | Source |
|---|---|---|
| MMLU | 4-choice | cais/mmlu |
| ARC-Challenge | Categorical | allenai/ai2_arc |
| GSM8K | Math + calculator | openai/gsm8k |
| HumanEval | Code execution disabled by default | openai/humaneval |
| SpellingBee | Tool use | Built-in templates |
| SmolTalk | Conversation | HuggingFaceTB/smol-smoltalk |
| CORE | ICL (DCLM) | Hellaswag, ARC, PIQA, Winogrande |
Test Suite
204 collected tests across 17 files cover model and attention semantics, all generation modes, optimizer safety, exact resume, evaluation, guarded execution, tokenizers, data/report utilities, sharding, and accelerator parity.
pixi run test # all tests pass on CPU, GPU, and TPU
Remote Execution
Kaggle TPU (CLI)
python -m scripts.kaggle_tpu_tests \
--kernel-id OWNER/flaxchat-tpu-tests --wait
GCP TPU (via tpuz)
from tpuz import TPU
tpu = TPU("my-tpu", accelerator="v6e-8")
tpu.up()
tpu.setup(extra_pip="flaxchat")
tpu.run("python -m scripts.pretrain --depth=12", sync=".")
Verified Results
See Verified accelerator results and its machine-readable provenance index for the canonical records. Only measurements linked there to immutable source, data, configuration, and hardware identities are treated as published flaxchat results.
Acknowledgments
This project is part of the 2026 Q1 TPU Sprint, supported by the Google AI Developer Programs team.
- Google AI Developer Programs for issuing GCP credits
- TPU Research Cloud (TRC) for providing free access to Cloud TPU v4, v5e, and v6e
- Kaggle for free TPU v5e access for prototyping
Built on nanochat, JAX, Flax, Optax, Orbax, tpuz, and the Kaggle CLI.
| View on GitHub | Documentation |
Blog Posts
- An Early TinyStories Training Experiment on 2xT4 GPUs — April 01, 2026 Historical note: the raw run record for this early experiment was not committed, so its quantitative tables have been withdrawn rather than treated as verified results. Current reproducible measurements live…
Posts
subscribe via RSS