Course · Math & optimizationPhase 04 · Study guide

Can you explain an optimizer’s behavior from the objective and data?

Core Foundation · Local CPULaptop CPU12 lessons (~15.5h)

Revision v0.18.1 ·

A learning curve is easier to understand when you can derive the update that produced it. Begin with vectors and residuals, then connect the gradient's direction to curvature and step size. A least-squares solve, a directional derivative and a finite-difference sweep provide different ways to challenge your answer. Read every optimizer experiment as a controlled comparison: what objective, data reduction, initialization and update budget stayed fixed? Work through one numerical step before interpreting a trajectory. Keep training fit, validation selection and final held-out reporting distinct as the examples become richer.

Preparation: State, randomness & control flow

Runtime: CPU

Check your starting point

A model predicts one value for each observation. Targets accidentally have an extra trailing singleton axis. Why might mean squared error still run?

Compare your reasoning

Broadcasting can construct a pairwise residual matrix instead of one residual per observation. Assert identical prediction and target shapes, then compute one residual by hand before reducing.

If this is unfamiliar, work through Linear algebra and loss intuition before attempting the project.

Build in stages

1. Audit the objective and derivative

Work through vectors, residuals, rank and directional derivatives. Use a linear solve and a finite-difference sweep as independent checks.

2. Connect dynamics to evaluation

Predict a stable versus divergent quadratic update, inspect stable likelihood arithmetic and separate fitting, validation selection and final test reporting.

3. Make optimizer comparisons fair

Count-weight unequal batches, preserve optimizer moments and schedule position, then change feature scale while retaining the prediction contract.

Try a changed condition

Make a prediction before opening the reasoning. Keep your answer and the evidence you would collect.

For the scalar loss L(w)=w2L(w)=w^2, predict the next error multipliers for learning rates 0.4, 1.0 and 1.1. Which curve can oscillate without converging?

Compare an approach

The update is wt+1=(1−2η)wtw_{t+1}=(1-2\eta)w_t. The factors are 0.20.2, −1-1 and −1.2-1.2. The middle case alternates at constant magnitude; the last grows. A plotting window can hide divergence when the starting value is tiny. Check the recurrence and plot absolute error on a logarithmic scale where nonzero.

Diagnose a likely failure

Symptom: Gradient descent diverges while all values initially remain finite.

Check next: Check the update sign, loss reduction, feature scales and largest quadratic curvature before trying a different seed.

Decide what is ready

Complete regression-audit, then optimizer-audit. Keep an analytic reference, stable/unstable trajectories, a resumed optimizer comparison and an untouched final evaluation split.

Where to put your project code

Write your code in: projects/optimizer-audit/my_optimization.py (copy projects/optimizer-audit/starter/optimization.py first: cp projects/optimizer-audit/starter/optimization.py projects/optimizer-audit/my_optimization.py).

Verify your implementation: python3 projects/optimizer-audit/tests/check.py --stage all --impl projects/optimizer-audit/my_optimization.py

Run the staged project checks on your implementation in projects/optimizer-audit/my_optimization.py and record your transfer experiment results in your portfolio.

Open A fitted regression model

What still needs further work

Constrained, nonsmooth and large-scale stochastic optimization are not covered comprehensively. Current ranking evidence is limited to declared synthetic objectives.