JAX Pathways

Course · Self-supervised pretraining: masked and contrastive learningPhase 17 · Study guide

What learning signal can unlabelled data provide?

Training Systems · Local CPULaptop CPU3 lessons (~5.3h)

Revision v0.7.0 ·

Train small masked-token, hidden-patch and paired-view models. Audit information leakage, reduction counts and pair identity before interpreting a falling objective.

Preparation: Transformers & language models

Runtime: Executed CPU synthetic objectives; large-scale and human-feedback evaluation remain separate

Start the phaseSee the project

Check your starting point

If an image reconstruction is perfect before training, what boundary should you inspect?

Compare your reasoning

Check whether hidden pixels entered the encoder or were copied into the prediction. A valid objective can still be defeated by leaked inputs.

If this is unfamiliar, work through Masked image modeling: reconstruct missing patches before attempting the project.

Build in stages

1. Masked language modeling: predict hidden tokens

Trace clean targets, corrupted IDs, attention axes and selected counts. Reproduce unequal-mask loss with a host calculation, derive the full logit gradient, and explain the final probability matrix.

2. Masked image modeling: reconstruct missing patches

Verify every patch on a rectangular RGB input, intervene on hidden pixels, and explain an ambiguous target that no deterministic decoder can reconstruct perfectly. Compare actual hidden targets with predictions.

3. Contrastive learning: views, positives and negatives

Derive a two-pair loss, change temperature without changing ranking, compare encoder gradients with finite differences, and diagnose broken identity or duplicate false negatives from the similarity matrix.

Try a changed condition

Make a prediction before opening the reasoning. Keep your answer and the evidence you would collect.

A batch contains duplicate semantic pairs. Should each duplicate automatically be a negative?

Compare an approach

No. Decide positives by source or semantic identity and use a compatible objective; the duplicate-pair exercise shows how a diagonal-only loss penalizes duplicates.

Diagnose a likely failure

Symptom: A low masked loss does not transfer to the downstream task.

Check next: Check target leakage, mask distribution, encoder-versus-projection features and a held-out downstream protocol.

Decide what is ready

Complete stages 1–3 of training-methods with independent loss references, real training curves, changed masks or pairs and explicit synthetic-data limits. Add the new lesson evidence: unequal-count gradient oracle, rectangular patch/intervention checks, irreducible ambiguity example and temperature/ranking comparison. Explain each saved figure without copying its caption.

Where to put your project code

Write your code in: projects/training-methods/my_methods.py (copy projects/training-methods/starter/methods.py first: cp projects/training-methods/starter/methods.py projects/training-methods/my_methods.py).

Verify your implementation: python3 projects/training-methods/tests/check.py --stage all --impl projects/training-methods/my_methods.py

Run the staged project checks on your implementation in projects/training-methods/my_methods.py and record your transfer experiment results in your portfolio.

Open An audited pretraining and adaptation objective toolkit

Open Trace a tiny bidirectional encoder from pretraining to retrieval

What still needs further work

Full BERT/ViT masked autoencoders, natural corpora, augmentation studies and downstream probe benchmarks remain extensions. The current labs execute small attention, linear reconstruction and embedding models.