Core Foundation · Local CPULaptop CPULesson 1 of 12 in Phase 04

Phase 04 / Math & optimization

Vectors, norms, and projections

Revision v0.7.0 · 2026-10-20

Lesson · about 60 minutes · CPU practice

Read a dot product geometrically and separate a vector into a projection and an orthogonal residual.

Phase study guide: preparation, milestones and project evidence

Before you start

Before this: State, randomness & control flow
Hardware: CPU

Download notebook

Other downloads

Python worked examples Lesson text

Setup help: Python, notebook, or Cloud TPU VM

Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions and accelerator environments require separate checks. The optional Cloud TPU specialization explains the target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.

Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.

macOS / Linux terminal · repository folder

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txt

PowerShell · repository folder

py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txt

After activation, run the complete worked example from that same folder:

macOS / Linux terminal · activated environment

python phases/04-optimization/05-vectors-norms-and-projections/code/main.py

PowerShell command · repository folder

.\.venv\Scripts\python.exe phases/04-optimization/05-vectors-norms-and-projections/code/main.py

For an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.

Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.

For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:

macOS / Linux terminal · activated course environment

python -m pip install jupyterlab
python -m jupyter lab

PowerShell command · repository folder

.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter lab

Select the downloaded optimization-05.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.

Check: Projection $[3,0]$ , residual $[0,4]$ . Both orthogonality and the length identity pass.

Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.

Phase 04.01 Stage 1 of 5 · Understand

1 Understand 2 Build & Run 3 Experiment 4 Exercise & Check 5 Evidence

What you will be able to do

Read a dot product geometrically and separate a vector into a projection and an orthogonal residual.
Read length and alignment separately
Find the closest point on a line
Use orthogonality as an independent check
Know when the formula is undefined

The problem

When a model multiplies features by weights, what is it measuring? Start with two arrows you can draw on paper. We will measure length, alignment and the part of one arrow that lies along the other. These ideas make regression and gradient directions easier to reason about.

The idea

Projecting a vector onto a direction keeps the part parallel to that direction. The residual is what the direction cannot explain. This small geometric idea will reappear in least squares, gradient components and constrained updates.

A projection explains what remains

Take $v=(3,4)$ and direction $u=(1,0)$ . The projection is $(3,0)$ , leaving residual $(0,4)$ . Their sum recovers $v$ , and the residual is perpendicular to $u$ . The original norm is $5$ , consistent with the right triangle.

Now replace $u$ by $(2,0)$ . The direction has not changed, so the projected vector should not change. The coefficient must adjust for the direction's squared norm. Forgetting that denominator makes the result depend on an arbitrary scaling of the direction.

Read the vector figure by following the projection and then the residual to the original endpoint. Use equal axis scale: otherwise a mathematically right angle may not look right on the page.

Read length and alignment separately

Let $u=(3,4)$ and $a=(1,0)$ . The length of $u$ is $5$ , and $u^\mathsf{T}a=3$ . The dot product is not a distance or a probability: reversing $a$ changes its sign. For nonzero vectors, dividing by both lengths gives the cosine of their angle, here $3/5$ . In higher dimensions we use exactly the same sum, even when we cannot draw all the axes.

\|u\|_2=\sqrt{\sum_i u_i^2},\qquad u^\mathsf{T}a=\sum_i u_i a_i

Find the closest point on a line

Points on the line through $a$ have the form $ca$ , where $c$ is a scalar. Choose the point nearest $u$ . Its coefficient is $c=(a^\mathsf{T}u)/(a^\mathsf{T}a)$ . For our axis direction, $c=3$ , so the projection is $p=(3,0)$ and the leftover is $r=u-p=(0,4)$ . Notice that the coefficient depends on how long we draw $a$ ; the projected point does not.

p=\frac{a^\mathsf{T}u}{a^\mathsf{T}a}a,\qquad r=u-p,\qquad a^\mathsf{T}r=0

Use orthogonality as an independent check

Orthogonal means perpendicular; algebraically, the dot product is zero. Because $p$ and $r$ are perpendicular, their squared lengths add to the squared length of $u$ : $9+16=25$ . We check both this identity and the leftover dot product. Matching a library projection to a copy of the same formula would give less independent evidence. For float32 calculations, a small absolute tolerance allows rounding near zero.

Know when the formula is undefined

The zero vector has no direction. Projecting onto it would divide by $0$ , and cosine similarity with it is also undefined. Our eager teaching helper raises a clear error for that input. A compiled application needs a separately defined policy, such as a validity mask or input validation before compilation. Adding a tiny constant silently changes the mathematical question; explain that policy if you choose it.

Projection and leftover form a right triangle

Predict before looking: Which part of $(3,4)$ is explained by the horizontal direction?

$Projection and leftover form a right triangle. Both axes are ordinary vector coordinates. The diagonal arrow goes from the origin to $u=(3,4)$. The horizontal arrow reaches the projection $p=(3,0)$, and the vertical arrow then goes from that point up to $u$. Follow the horizontal and vertical arrows tip to tail: together they reproduce the diagonal arrow. Although the residual arrow is drawn starting at $(3,0)$, its displacement is $r=(0,4)$, not $(3,4)$.$

Plot from a recorded CPU computation · JAX 0.9.2 · 1 CPU device

Read the figure

Both axes are ordinary vector coordinates. The diagonal arrow goes from the origin to $u=(3,4)$ . The horizontal arrow reaches the projection $p=(3,0)$ , and the vertical arrow then goes from that point up to $u$ .

Follow the horizontal and vertical arrows tip to tail: together they reproduce the diagonal arrow. Although the residual arrow is drawn starting at $(3,0)$ , its displacement is $r=(0,4)$ , not $(3,4)$ .

Connect it to the computation

We projected onto the horizontal direction. The residual is perpendicular to that direction, so its dot product with $(1,0)$ is zero. This right angle explains why moving the projection along the horizontal line cannot remove the remaining vertical error.

The lengths form a familiar check: $\lVert p\rVert=3$ , $\lVert r\rVert=4$ , and $\lVert u\rVert=5$ , with $3^2+4^2=5^2$ . Projection separates the part represented by the chosen direction from the part that direction cannot represent.

Open full-size figure Download data and provenance

The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.

Where to put & run this code: Work in phases/04-optimization/05-vectors-norms-and-projections/code/main.py (or downloaded exercises/optimization-05.py / exercises/optimization-05.ipynb) and execute from the repository root with python3 phases/04-optimization/05-vectors-norms-and-projections/code/main.py.

1 Draw the two vectors

Create main.py in your activated CPU course environment. Add the vectors and verify their lengths before adding a projection.

main.py · append build step 1

# Step 1 — Draw the two vectors: The length is 5; the component along the horizontal unit direction...
import jax.numpy as jnp
u = jnp.array([3.0, 4.0])
a = jnp.array([1.0, 0.0])
assert jnp.allclose(jnp.linalg.norm(u), 5.0)
assert jnp.allclose(jnp.dot(u, a), 3.0)

Code building blocks in this step

jnp.array(values, dtype=...) — Constructs an immutable device-backed JAX array from Python/NumPy values.
jnp.allclose(actual, expected, rtol=..., atol=...) — Checks that two arrays match elementwise within floating-point tolerance.

The length is $5$ ; the component along the horizontal unit direction is $3$ .

2 Build the projection

Append this eager helper. Keep the input check outside any jitted function in this exercise.

main.py · append build step 2

# Step 2 — Build the projection: The leftover r contains what the chosen direction cannot explain.
def project(u, a):
    denominator = jnp.dot(a, a)
    # Guard input contract (`float(denominator) == 0.0`) and fail fast if violated.
    if float(denominator) == 0.0:
        raise ValueError('Projection needs a nonzero direction')
    return jnp.dot(a, u) / denominator * a
p = project(u, a)
r = u - p

Code building blocks in this step

project(...) — Call project with your updated parameters or inputs from this lesson's workspace.
jnp.dot(...) — Call jnp.dot with your updated parameters or inputs from this lesson's workspace.

The leftover $r$ contains what the chosen direction cannot explain.

3 Check with geometry

Append these assertions and run python main.py. Compare with your drawing before reading the experiment.

main.py · append build step 3

# Step 3 — Check with geometry: You should see [3,0] and [0,4]; the squared lengths sum to 25.
assert jnp.allclose(p, jnp.array([3.0, 0.0]))
assert jnp.allclose(jnp.dot(a, r), 0.0, atol=1e-06)
assert jnp.allclose(jnp.dot(p, p) + jnp.dot(r, r), jnp.dot(u, u))
print('projection / residual:', p, r)

Code building blocks in this step

jnp.array(values, dtype=...) — Constructs an immutable device-backed JAX array from Python/NumPy values.
jnp.allclose(actual, expected, rtol=..., atol=...) — Checks that two arrays match elementwise within floating-point tolerance.

You should see $[3,0]$ and $[0,4]$ ; the squared lengths sum to $25$ .

View the cumulative build in main.py

These blocks append in order to the same file.

main.py · cumulative Python build

# Step 1 — Draw the two vectors: The length is 5; the component along the horizontal unit direction...
import jax.numpy as jnp
u = jnp.array([3.0, 4.0])
a = jnp.array([1.0, 0.0])
assert jnp.allclose(jnp.linalg.norm(u), 5.0)
assert jnp.allclose(jnp.dot(u, a), 3.0)

# Step 2 — Build the projection: The leftover r contains what the chosen direction cannot explain.
def project(u, a):
    denominator = jnp.dot(a, a)
    # Guard input contract (`float(denominator) == 0.0`) and fail fast if violated.
    if float(denominator) == 0.0:
        raise ValueError('Projection needs a nonzero direction')
    return jnp.dot(a, u) / denominator * a
p = project(u, a)
r = u - p

# Step 3 — Check with geometry: You should see [3,0] and [0,4]; the squared lengths sum to 25.
assert jnp.allclose(p, jnp.array([3.0, 0.0]))
assert jnp.allclose(jnp.dot(a, r), 0.0, atol=1e-06)
assert jnp.allclose(jnp.dot(p, p) + jnp.dot(r, r), jnp.dot(u, u))
print('projection / residual:', p, r)

Run the example

python3 phases/04-optimization/05-vectors-norms-and-projections/code/main.py

About this command

This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.

Inspect the complete assembled example

main.py · complete Python example

# Step 1 — Draw the two vectors: The length is 5; the component along the horizontal unit direction...
import jax.numpy as jnp
u = jnp.array([3.0, 4.0])
a = jnp.array([1.0, 0.0])
assert jnp.allclose(jnp.linalg.norm(u), 5.0)
assert jnp.allclose(jnp.dot(u, a), 3.0)

# Step 2 — Build the projection: The leftover r contains what the chosen direction cannot explain.
def project(u, a):
    denominator = jnp.dot(a, a)
    # Guard input contract (`float(denominator) == 0.0`) and fail fast if violated.
    if float(denominator) == 0.0:
        raise ValueError('Projection needs a nonzero direction')
    return jnp.dot(a, u) / denominator * a
p = project(u, a)
r = u - p

# Step 3 — Check with geometry: You should see [3,0] and [0,4]; the squared lengths sum to 25.
assert jnp.allclose(p, jnp.array([3.0, 0.0]))
assert jnp.allclose(jnp.dot(a, r), 0.0, atol=1e-06)
assert jnp.allclose(jnp.dot(p, p) + jnp.dot(r, r), jnp.dot(u, u))
print('projection / residual:', p, r)

Expected result: Projection $[3,0]$ , residual $[0,4]$ . Both orthogonality and the length identity pass.

Recorded reference execution

Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.

2026-10-20T10:45:33.435928+00:00 · Python 3.14.3 · JAX 0.9.2 · 1 CPU device

projection / residual: [3. 0.] [0. 4.]
projection / residual: [3. 0.] [0. 4.]
PASS: optimization-05

Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.

Rescale the direction

Rescale the direction · Python experiment

# Experiment — Rescale the direction: A line does not change when its nonzero direction is rescaled.
assert jnp.allclose(project(u, 2 * a), p)
assert jnp.allclose(project(u, -a), p)
assert jnp.allclose(jnp.dot(u, 2 * a), 2 * jnp.dot(u, a))

Expected: The projection stays fixed; the raw dot product doubles.

A line does not change when its nonzero direction is rescaled. A dot product alone does change.

Rotate the line

Rotate the line · Python experiment

# Experiment — Rotate the line: Changing direction changes what can be explained; merely...
diagonal = jnp.array([1.0, 1.0])
p2 = project(u, diagonal)
assert jnp.allclose(p2, jnp.array([3.5, 3.5]))
assert jnp.allclose(jnp.dot(diagonal, u - p2), 0.0, atol=1e-06)

Expected: Projection $[3.5,3.5]$ , residual $[-0.5,0.5]$ .

Changing direction changes what can be explained; merely changing its length does not.

Hands-On Exercise · Modify & Verify Run in the declared lesson environment

Make it yours

Where to put your code: Append your solution at the bottom of phases/04-optimization/05-vectors-norms-and-projections/code/main.py (or inside the Make it yours cell of exercises/optimization-05.ipynb) and run python3 phases/04-optimization/05-vectors-norms-and-projections/code/main.py to verify your prediction before opening the reference solution.

Project $u=(2,-1,2)$ onto $a=(1,0,1)$ . Check the residual by hand first.

Show a writing guide and starter scaffold

How to write this exercise code: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

jnp.array(values, dtype=...) — Constructs an immutable device-backed JAX array from Python/NumPy values.
jnp.allclose(actual, expected, rtol=..., atol=...) — Checks that two arrays match elementwise within floating-point tolerance.

2. Step-by-step implementation plan:

Set up the changed input or configuration parameters requested in the prompt.
Run the target function or transformation and assert that the output matches your prediction.

Starter code scaffold · copy into your file & fill in the TODOs

# Exercise solution: Project u=(2,-1,2) onto a=(1,0,1).
u3 = jnp.array(...)  # TODO: compute u3
a3 = jnp.array(...)  # TODO: compute a3
p3 = project(...)  # TODO: compute p3
assert jnp.allclose(p3, jnp.array([2.0, 0.0, 2.0]))  # TODO: complete assertion check
assert jnp.allclose(jnp.dot(a3, u3 - p3), 0.0)  # TODO: complete assertion check

Reveal the reference solution

Reference solution

# Exercise solution: Project u=(2,-1,2) onto a=(1,0,1).
u3 = jnp.array([2.0, -1.0, 2.0])
a3 = jnp.array([1.0, 0.0, 1.0])
p3 = project(u3, a3)
assert jnp.allclose(p3, jnp.array([2.0, 0.0, 2.0]))
assert jnp.allclose(jnp.dot(a3, u3 - p3), 0.0)

Handle an undefined direction

Transfer / diagnosis

Show that a zero direction produces a clear input error instead of a misleading projection.

Show a writing guide and starter scaffold

How to write this practice solution: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

direction(...) — Call direction with your updated parameters or inputs from this lesson's workspace.
project(...) — Call project with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

2. Step-by-step implementation plan:

Run the boundary check and catch the expected exception:

Starter code scaffold · copy into your file & fill in the TODOs

# Handle an undefined direction (Transfer / diagnosis): Rejecting an undefined input is part of the function contract.
# Run the boundary check and catch the expected exception:
try:
    project(u, jnp.zeros_like(a))
except ValueError as exc:
    assert 'nonzero'  # TODO: complete assertion check
else:
    raise AssertionError('Zero direction was accepted')

Show a hint

Use a try/except block; a silent successful result should fail the check.

Reveal reference solution and reasoning

Handle an undefined direction · Python solution

# Handle an undefined direction (Transfer / diagnosis): Rejecting an undefined input is part of the function contract.
# Run the boundary check and catch the expected exception:
try:
    project(u, jnp.zeros_like(a))
except ValueError as exc:
    assert 'nonzero' in str(exc)
else:
    raise AssertionError('Zero direction was accepted')

Rejecting an undefined input is part of the function contract.

Use two perpendicular directions

Transfer / diagnosis

Reconstruct $u$ from projections onto $(1,1)$ and $(1,-1)$ . Would adding independent projections still work for arbitrary nonorthogonal directions?

Show a writing guide and starter scaffold

How to write this practice solution: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

jnp.array(values, dtype=...) — Constructs an immutable device-backed JAX array from Python/NumPy values.
jnp.allclose(actual, expected, rtol=..., atol=...) — Checks that two arrays match elementwise within floating-point tolerance.

2. Step-by-step implementation plan:

Set up the changed input or configuration parameters requested in the prompt.
Run the target function or transformation and assert that the output matches your prediction.

Starter code scaffold · copy into your file & fill in the TODOs

# Use two perpendicular directions (Transfer / diagnosis): Adding separate projections reconstructs a vector in an...
a1 = jnp.array(...)  # TODO: compute a1
a2 = jnp.array(...)  # TODO: compute a2
assert jnp.allclose(project(u, a1) + project(u, a2), u)  # TODO: complete assertion check
assert not jnp.allclose(project(u, a) + project(u, a1), u)  # TODO: complete assertion check

Show a hint

Check the directions have zero dot product. Nonorthogonal directions can count the same component twice.

Reveal reference solution and reasoning

Use two perpendicular directions · Python solution

# Use two perpendicular directions (Transfer / diagnosis): Adding separate projections reconstructs a vector in an...
a1 = jnp.array([1.0, 1.0])
a2 = jnp.array([1.0, -1.0])
assert jnp.allclose(project(u, a1) + project(u, a2), u)
assert not jnp.allclose(project(u, a) + project(u, a1), u)

Adding separate projections reconstructs a vector in an orthogonal basis. Correlated feature directions require a joint least-squares solve.

Diagnose the result

If the leftover is not perpendicular, check the denominator: it is $a^\mathsf{T}a$ , not the length of $a$ . If cosine exceeds its expected range, check normalization and rounding; first reject zero-length inputs.

Carry forward

A line does not change when its nonzero direction is rescaled. A dot product alone does change.
Changing direction changes what can be explained; merely changing its length does not.

Keep your evidence

Download evidence template

Keep the hand-computed projection, perpendicular-residual check, rescaling experiment, zero-direction repair and changed-basis calculation.

Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.

Saved in this browser. Export or back up your portfolio anytime in My learning.

Save work for this lesson

Verified on local CPU. Stage to a Cloud TPU VM via Cloud TPU setup to record accelerator execution receipts (TODO: add automated Cloud TPU CI execution receipts).

Vectors, norms, and projections

What you will be able to do

The problem

The idea

A projection explains what remains

Read length and alignment separately

Find the closest point on a line

Use orthogonality as an independent check

Know when the formula is undefined

Projection and leftover form a right triangle

Read the figure

Connect it to the computation

1 Draw the two vectors

2 Build the projection

3 Check with geometry

Run the example

Recorded reference execution

Rescale the direction

Rotate the line

Make it yours

Handle an undefined direction

Use two perpendicular directions

Check your understanding

Diagnose the result

Carry forward

Keep your evidence

Primary references