flaxchat

flaxchat

A minimal, end-to-end LLM training harness for Google Cloud TPU pods, built on JAX/Flax NNX.

Port of nanochat to the JAX ecosystem with full feature parity plus speculative decoding.

Part of the 2026 Q1 TPU Research Sprint, supported by the Google AI Developer Programs team.


Quick Start

pixi install
pixi run test
pixi run test-e2e
pixi run -- python -m scripts.run_tinystories --layers=8 --pretrain-steps=5000

Core Features

GPT Architecture (all nanochat features)

  • Rotary Embeddings (RoPE), Group-Query Attention (GQA), QK Normalization
  • ReLU^2 MLP, Value Embeddings (ResFormer), Sliding Window (SSSL pattern)
  • Per-layer Residual Scaling, Smear, Backout, Logit Soft-capping
  • Gradient Checkpointing (nnx.remat, dots_saveable)

Optimizer

  • Mixed Muon + AdamW via optax.multi_transform
  • Muon: Polar Express orthogonalization + NorMuon variance reduction
  • Per-group LR, betas, weight decay. Warmup -> constant -> warmdown schedule

Inference Engine (4 modes)

Mode Function Speed Description
Padded generate() Measured per run Simple, for debugging
KV-cached generate_with_cache() Measured per run Python loop with KV cache
Fully JIT generate_fast() Measured per run jax.lax.while_loop, no Python overhead
Speculative generate_speculative() Benchmark per pairing Batched main-model verification

Tool Use (streaming)

engine = Engine(model, tokenizer)
for tokens, masks in engine.generate(prompt_ids, num_samples=3):
    # Automatic calculator; generated Python is disabled by default
    pass

Guarded Code Execution

from flaxchat.execution import execute_code
result = execute_code("print(sum(range(10)))", timeout=5.0, trusted=True)
# ExecutionResult(success=True, stdout="45\n")

This is a best-effort reliability guard for reviewed code, not a security sandbox. Untrusted execution is disabled unless an external isolation backend is explicitly configured.

Parallelism (default, not optional)

mesh = compute_init()  # auto mesh over ALL devices
# Data parallel, FSDP, multi-host — all automatic via JAX SPMD

Depth-Based Config

config = FlaxChatConfig.from_depth(depth=12)
# -> 12 layers, 768 dims, 6 heads, ~79M params

Pipeline

Stage Script Description
Tokenizer scripts/tok_train.py Train BPE tokenizer
Pretrain scripts/pretrain.py Pretrain on ClimbMix-400B / TinyStories
SFT scripts/sft.py Supervised fine-tuning
RL scripts/rl.py GRPO/REINFORCE on GSM8K
Eval scripts/eval.py MMLU, ARC, GSM8K, HumanEval, CORE
Chat scripts/chat_web.py FastAPI WebSocket UI
Export scripts/convert_to_tflite.py LiteRT/TFLite for edge

Evaluation Tasks

Task Type Source
MMLU 4-choice cais/mmlu
ARC-Challenge Categorical allenai/ai2_arc
GSM8K Math + calculator openai/gsm8k
HumanEval Code execution disabled by default openai/humaneval
SpellingBee Tool use Built-in templates
SmolTalk Conversation HuggingFaceTB/smol-smoltalk
CORE ICL (DCLM) Hellaswag, ARC, PIQA, Winogrande

Test Suite

204 collected tests across 17 files cover model and attention semantics, all generation modes, optimizer safety, exact resume, evaluation, guarded execution, tokenizers, data/report utilities, sharding, and accelerator parity.

pixi run test  # all tests pass on CPU, GPU, and TPU

Remote Execution

Kaggle TPU (CLI)

python -m scripts.kaggle_tpu_tests \
  --kernel-id OWNER/flaxchat-tpu-tests --wait

GCP TPU (via tpuz)

from tpuz import TPU
tpu = TPU("my-tpu", accelerator="v6e-8")
tpu.up()
tpu.setup(extra_pip="flaxchat")
tpu.run("python -m scripts.pretrain --depth=12", sync=".")

Verified Results

See Verified accelerator results and its machine-readable provenance index for the canonical records. Only measurements linked there to immutable source, data, configuration, and hardware identities are treated as published flaxchat results.

Acknowledgments

This project is part of the 2026 Q1 TPU Sprint, supported by the Google AI Developer Programs team.

Built on nanochat, JAX, Flax, Optax, Orbax, tpuz, and the Kaggle CLI.

View on GitHub Documentation

Blog Posts

  • An Early TinyStories Training Experiment on 2xT4 GPUs — April 01, 2026 Historical note: the raw run record for this early experiment was not committed, so its quantitative tables have been withdrawn rather than treated as verified results. Current reproducible measurements live…

Posts

subscribe via RSS