Multimodal & Adaptation

Training from scratch is expensive. This track is about reusing and adapting models — attaching small trainable adapters to a frozen network, and aligning models across different modalities (like images and text).

What you'll build

• LoRA Fine-Tuning — freeze a full model and train only tiny low-rank adapters (nnx.LoRALinear), updating well under 1% of the parameters. Uses nnx.Optimizer(wrt=nnx.LoRAParam) to scope training to just the adapters.
• CLIP (toy) — align an image encoder and a text encoder in a shared embedding space with a symmetric contrastive loss.

Prerequisites

You should have built a transformer (the model we adapt) and understand Flax NNX state and parameter filtering.

Next steps

• Build LoRA Fine-Tuning.
• Contrast with Knowledge Distillation, a different efficiency approach (compress into a smaller student).