JAX PathwaysCheck your starting point
A query has equal scores for three allowed keys. What are the attention weights, and what happens if one key is masked?
Compare your reasoning
The weights are equal thirds. Masking one key leaves equal halves on the two allowed keys and zero on the masked key. Normalize over keys, not queries.
If this is unfamiliar, work through Attention from small pieces before attempting the project.
Build in stages
1. Audit attention and targets
Check one attention row by hand, token/target shifts, padding denominators and segment boundaries. Perturb future tokens and inspect earlier outputs.
2. Train, resume and decode one small model
Preserve residual shapes and sensitive arithmetic, compare the next resumed update and state the decoding policy. Continue into the text harness for cache/full-prefix parity.
Try a changed condition
Make a prediction before opening the reasoning. Keep your answer and the evidence you would collect.
Two documents are packed into one sequence. Causal masking passes, but one document still changes the other’s predictions. What is missing?
Compare an approach
Causality only blocks future positions. Require matching segment identities too, align targets within each document, and exclude padding or invalid boundaries from the loss. Test a change to the other document while holding the target document fixed.
Diagnose a likely failure
Symptom: Padding produces NaNs in attention.
Check next: Count permitted keys in every query row. A fully masked softmax needs an explicit invalid-query policy; its output and loss must not become valid training evidence.
Decide what is ready
Use text-harness stages 1–4. Keep independent attention arithmetic, future-token invariance, resumed Adam state and full-prefix/cache parity with declared precision.
Where to put your project code
Write your code in: projects/text-harness/my_model.py (copy projects/text-harness/starter/model.py first: cp projects/text-harness/starter/model.py projects/text-harness/my_model.py).
Verify your implementation: python3 projects/text-harness/tests/check.py --stage all --impl projects/text-harness/my_model.py
Run the staged project checks on your implementation in projects/text-harness/my_model.py and record your transfer experiment results in your portfolio.
Open Trace a tiny bidirectional encoder from pretraining to retrieval
What still needs further work
Real-corpus curation, tokenizer tradeoff studies, long-context evaluation and live generative quality need further labs beyond the tiny byte-model harness.