Systems Specialization · CPU ReferenceCPU reference; target qualification separateLesson 5 of 8 in Phase 15

Phase 15 / Deployment, interoperability & edge AI

Choose weight, activation and accumulation precision

Revision v0.7.0 · 2026-10-20

Lesson · about 110 minutes · CPU practice

Compare floating-point policies, weight-only quantization and calibrated integer activations.

Phase study guide: preparation, milestones and project evidence

Before you start

Before this: Export a computation and verify its serving contract
Hardware: CPU; optional framework and edge-device labs have separate requirements

Download notebook

Other downloads

Python worked examples Lesson text

Setup help: Python, notebook, or Cloud TPU VM

Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions and accelerator environments require separate checks. The optional Cloud TPU specialization explains the target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.

Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.

macOS / Linux terminal · repository folder

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txt

PowerShell · repository folder

py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txt

After activation, run the complete worked example from that same folder:

macOS / Linux terminal · activated environment

python phases/15-deployment/05-weight-activation-and-accumulation-precision/code/main.py

PowerShell command · repository folder

.\.venv\Scripts\python.exe phases/15-deployment/05-weight-activation-and-accumulation-precision/code/main.py

For an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.

Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.

For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:

macOS / Linux terminal · activated course environment

python -m pip install jupyterlab
python -m jupyter lab

PowerShell command · repository folder

.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter lab

Select the downloaded deployment-05.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.

Check: FP32, FP16 and BF16 comparisons are finite with error below $0.05$ for this fixture. INT8/INT4 reconstruction errors obey a half-scale bound; these are simulated weight-only computations.

Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.

Phase 15.05 Stage 1 of 5 · Understand

1 Understand 2 Build & Run 3 Experiment 4 Exercise & Check 5 Evidence

What you will be able to do

Compare FP32, FP16 and BF16 rounding without confusing storage with accumulation.
Implement per-output-channel symmetric INT8 and INT4 weight quantization.
Calibrate activations and diagnose clipping on unseen ranges.
Explain FP64, FP8, weight-only, full integer and mixed precision policies.

The problem

Someone asks you for an “8-bit model.” A helpful first question is: 8-bit weights, activations, or both? We’ll separate those choices, compare them with a float32 reference, and watch what happens when an input falls outside the calibrated range. You’ll leave with a precision policy you can explain, not just a dtype label.

The idea

A precision policy specifies how weights, activations, accumulations and outputs are represented. Quantization adds scales, zero-points and rounding/clipping rules. Track those units through the computation instead of describing the whole model with one dtype label.

Follow the units through integer arithmetic

For an illustrative scale $s=0.5$ and zero-point $z=3$ , the real value $1$ maps to integer $q=5$ , since $q=\operatorname{round}(1/0.5)+3$ . Reconstructing gives $0.5(5-3)=1$ . Real zero maps to the zero-point, not necessarily integer zero.

In a dot product, centered integer products accumulate in an appropriately wide type. The accumulator's scale comes from the input and weight scales. Bias must use compatible units before the output is rescaled or requantized.

The error plots compare specific representation policies. Read weight, activation and accumulation choices separately. Per-channel scales also require a declared axis. Numerical error, downstream quality and target-runtime speed need separate evidence.

Follow affine integer codes into centered products, wide accumulation and compatible bias units, then output rescaling. Exact accumulator dtype and per-channel scale axes belong to the declared runtime policy. These arrows describe arithmetic dependencies, not a specific hardware kernel. — Architecture and dataflow mechanism diagram.

Write the policy before choosing a dtype

Use separate fields for stored weights, input/intermediate activations, multiply operands, accumulator, output and optimizer/master weights. FP32 is a useful baseline. FP64 can help sensitive numerical work; in JAX enable jax_enable_x64 before creating arrays and account for target support and cost. FP16 has a narrower exponent range than BF16, while BF16 gives up fraction precision for range. Keep reductions, normalization statistics and sensitive losses in FP32 when needed. The example rounds operands to each format, then explicitly returns them to FP32 for multiplication; it isolates representation error rather than proving low-precision kernel behavior.

Mixed precision during training

A Keras mixed_float16 or mixed_bfloat16 policy generally keeps variables in FP32 while using lower-precision computations. Set the policy before constructing layers and inspect variable_dtype and compute_dtype. FP16 training may need loss scaling to avoid gradient underflow; unscale before clipping and updating, reject nonfinite updates, and keep optimizer/master state at suitable precision. BF16 has wider range but still incurs rounding. In explicit JAX code, cast intentionally at operation boundaries and test the full update. Merely changing the parameter storage dtype is not a complete mixed precision implementation.

Quantize weights with scale metadata

Let $b$ be the number of bits and $q_{\max}=2^{b-1}-1$ the largest positive code. For each weight column, choose scale $s=\max(|w|)/q_{\max}$ . Dividing a weight by $s$ moves it onto the integer grid; rounding chooses the nearest code, and clipping keeps the code in range. Reconstruct the approximate weight with $\widehat w=sq$ .

Our Dense kernel’s output channels lie along axis $1$ , so each column gets its own scale. This preserves resolution when columns have different ranges. For a zero column, use scale $1$ and codes $0$ . Without clipping, rounding changes a value by at most $s/2$ . Smaller quantization groups can improve local resolution, but each group also needs scale metadata.

Weight-only and full integer are different deployments

W4A16 and W8A16 usually describe compressed weights with floating activations; a runtime may dequantize before multiplication or use a specialized mixed kernel. Our 4-bit codes live in int8 arrays and are not nibble-packed, so they do not realize 4-bit memory savings. W8A8 quantizes activations too. An affine activation reconstructs as $a=s(q-z)$ , where $q$ is its integer code, $s$ is its scale, and $z$ is the zero point; calibration estimates its range from representative training/calibration data, never the held-out evaluation set. Static quantization fixes ranges; dynamic quantization estimates activation ranges at runtime. Accumulation commonly uses INT32 for integer kernels; bias and requantization must use compatible scales.

FP8 and other low-bit formats require a target contract

FP8 E4M3 and E5M2 differ in range and precision; their names are not interchangeable. Scaling granularity, saturation behavior and kernel support matter. Do not treat a successful CPU dtype cast as evidence of FP8 acceleration. INT4, UINT8, INT16 activation schemes and newer sub-byte floating formats also depend on runtime/operator support. For ONNX Runtime or LiteRT, inspect the format, operator, quantization scheme and execution provider together. If a supported PTQ policy loses too much task accuracy, evaluate quantization-aware training rather than silently relaxing acceptance criteria.

Evaluate quality separately from size and speed

Compare the original and converted models on held-out examples, including low-margin decisions and uncommon ranges. Record absolute/relative error with a sensible near-zero floor, task metrics, clipping rates and acceptance thresholds chosen in advance. Estimate weight storage as ceil(parameter_count*bits/8) plus metadata, but measure the real artifact and peak runtime memory. Activation buffers, KV caches, workspace and fallback copies can dominate. Benchmark on the intended device after verifying which kernels actually execute.

Follow one value onto the quantization grid

A scale $s$ tells us how far apart the representable values are. For $s=0.1$ , a weight of $0.26$ rounds to integer code $3$ and reconstructs as $0.3$ . The error is $0.04$ , less than half a grid step. If a value lies outside the supported range, clipping can make the error much larger. Our per-channel code computes a separate scale for each output column and treats a zero column specially.

\begin{aligned}q_{\max}&=2^{b-1}-1\\q&=\operatorname{clip}\!\left(\operatorname{round}(w/s),-q_{\max},q_{\max}\right)\\\widehat w&=sq,\qquad |w-\widehat w|\le s/2\quad\text{without clipping}\end{aligned}

Follow an affine output through every numeric unit

Let an activation code be $q_x$ , its zero point $z_x$ , and its positive scale $s_x$ . The represented value is $s_x(q_x-z_x)$ . Subtract the zero point after widening the codes: raw INT8 arithmetic can overflow before accumulation. With symmetric weights, each output channel has weight zero point zero and scale $s_{w,j}$ . Its accumulator therefore has units $s_xs_{w,j}$ , so round the bias into those same units before adding it. In the worked experiment, corrected products plus bias give accumulators $5$ and $15$ ; their real outputs are $0.625$ and $0.9375$ . Requantization maps these values onto the output grid. A small output scale offers fine spacing but can also make the representable range too narrow. The second plot separates forgetting the input zero point from clipping at the output; these are different bugs with different repairs.

a_j=\sum_i(q_{x,i}-z_x)q_{w,ij}+\operatorname{round}\!\left(\frac{b_j}{s_xs_{w,j}}\right),\qquad q_{y,j}=\operatorname{clip}\!\left(\operatorname{round}\!\left(\frac{s_xs_{w,j}}{s_y}a_j\right)+z_y,-128,127\right)

Distinguish storage savings from an error budget

For a dense matrix with $K$ input features and $N$ output features, FP32 weights occupy $4KN$ bytes before container metadata. Ideally packed four-bit weights occupy $\lceil KN/2\rceil$ bytes, but each output channel may also need a floating-point scale. If scales are FP32, add $4N$ bytes. Biases, alignment and runtime workspace are additional terms.

For this tiny $(3,2)$ matrix, FP32 needs $24$ weight bytes. A packed four-bit representation plus two FP32 scales would need $3+8=11$ bytes before other metadata. The teaching implementation stores its codes in int8 arrays, so it uses $6+8=14$ bytes instead. Report the representation you actually measured. Neither byte count says which kernel will execute or how long a request will take.

Follow rounding error through the next multiplication

A weight value rounded to its nearest grid point has error at most half a scale when it is within the represented range. To see why that does not directly bound the output by half a scale, write the output error as a sum of input-weight error products. Large inputs can amplify small weight errors; cancellation may reduce the observed error but should not be assumed in the bound.

For input $[2,-1,0.5]$ and weight errors $[0.1,-0.1,0.1]$ , the signed output error is $0.35$ . The sum of absolute products is also $0.35$ , so the bound can be attained. Activation clipping is different: a value outside the represented range can lose much more than half a scale. Keep a clipping count alongside your rounding and task-quality measurements.

|\Delta y_j|=\left|\sum_i x_i\Delta W_{ij}\right|\leq\sum_i |x_i|\,|\Delta W_{ij}|

Precision choices trade representation for output error

Predict before looking: Must two low-precision formats have the same error?

Read the figure

The horizontal categories are precision policies; the vertical axis is maximum absolute output error against the FP32 reference on the same small input. Lower bars mean closer numerical agreement, not faster execution. The FP32 baseline is zero by construction.

The errors are approximately $0.00105$ for FP16 rounding, $0.00617$ for BF16 rounding, $0.00803$ for W8 A32, and $0.146$ for W4 A32. The four-bit weight simulation has about eighteen times the output error of the eight-bit simulation in this example. These are output units, not percentages or classification error rates.

The lower panel uses a separate exactly representable affine fixture, with output channel on the horizontal axis and real output value on the vertical axis. Correct values are $0.625$ and $0.9375$ . Omitting the input zero point lowers them to $-0.125$ and $0.5625$ ; this is systematic offset error, not random rounding. The narrow output grid makes both bars equal $0.52734375$ , the largest representable output under that policy. Equal bars here reveal saturation, not equal underlying predictions.

Connect it to the computation

For the floating-point policies, both inputs and weights are rounded, then converted to FP32 for accumulation. FP16 has finer precision than BF16 around these particular values; BF16’s wider exponent range is not being tested here. The weight-only policies keep activations in FP32 and reconstruct quantized weights before multiplication.

Four-bit quantization uses fewer representable weight levels, so its larger rounding steps can create larger output discrepancies. The code uses per-output-column scales and checks an independent error bound. This is simulated quantization using ordinary CPU arrays, not a measurement of packed four-bit storage or native low-bit kernels. Accuracy, memory, and latency must each be evaluated on the intended workload and runtime.

Use the lower panel to decide what to inspect: zero-point subtraction and bias units for an offset mismatch; output range and clipping counts for saturation. The upper panel compares precision policies on the original fixture, while the lower panel isolates operator bookkeeping on a different fixture. Neither measures latency.

Open full-size figure Download data and provenance

The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.

Where to put & run this code: Work in phases/15-deployment/05-weight-activation-and-accumulation-precision/code/main.py (or downloaded exercises/deployment-05.py / exercises/deployment-05.ipynb) and execute from the repository root with python3 phases/15-deployment/05-weight-activation-and-accumulation-precision/code/main.py.

1 Write the full-precision reference

Create main.py in your lesson workspace and run it with the active course Python environment. Keep the input and kernel in FP32 and establish an independently computed dot product.

main.py · append build step 1

# Step 1 — Write the full-precision reference: The top-left output is -1.437.
import numpy as np
import jax.numpy as jnp
x = np.array([[.3, -1.2, 2.1], [1.1, .2, -.8]], np.float32)
w = np.array([[.11, -1.8], [.7, .2], [-.3, 2.4]], np.float32)
# Perform matrix / vector contraction (`@`) to compute `reference`.
reference = x @ w

np.testing.assert_allclose(reference[0,0], -1.437, atol=1e-6)

Code building blocks in this step

jnp.allclose(actual, expected, rtol=..., atol=...) — Checks that two arrays match elementwise within floating-point tolerance.

The top-left output is $-1.437$ . This anchors the error comparison to a named value before introducing a lower-precision policy.

2 Round inputs and weights, then accumulate explicitly

Append this block to the same main.py and rerun the whole file. Compare FP32, FP16 and BF16 representations while returning to FP32 before multiplication.

main.py · append build step 2

# Step 2 — Round inputs and weights, then accumulate explicitly: The printed errors isolate the consequence of representational...
def mixed_forward(x, w, dtype):
    # Demonstrate rounding inputs/weights with explicit float32 accumulation.
    a = jnp.asarray(x, dtype).astype(jnp.float32)
    # Create device-backed JAX array `b`.
    b = jnp.asarray(w, dtype).astype(jnp.float32)
    return np.asarray(a @ b)
for dtype in (jnp.float32, jnp.float16, jnp.bfloat16):
    y = mixed_forward(x, w, dtype)
    # Aggregate array values to compute `error`.
    error = float(np.max(np.abs(y-reference)))
    # Confirm that all computed values remain finite (no NaN or Inf).
    assert np.isfinite(y).all() and error < .05
    print(str(dtype), "max absolute error", error)

Code building blocks in this step

jnp.isfinite(x) — Returns a boolean mask verifying that no element is NaN or Inf.

The printed errors isolate the consequence of representational rounding in this example. This does not benchmark a native low-precision matrix kernel.

3 Construct signed integer codes and channel scales

Append this block to the same main.py and rerun the whole file. Quantize per output channel and inspect the all-zero-channel case.

main.py · append build step 3

# Step 3 — Construct signed integer codes and channel scales: An all-zero channel uses scale 1 and zero codes, so reconstruction...
def quantize_weights(weights, bits):
    # Guard input contract (`bits not in (4, 8)`) and fail fast if violated.
    if bits not in (4, 8):
        raise ValueError("use signed 4 or 8 bit symmetric quantization")
    limit = 2 ** (bits - 1) - 1
    # Reduce along axis=0 to compute `maxima`.
    maxima = np.max(np.abs(weights), axis=0, keepdims=True)
    scale = np.where(maxima == 0, 1., maxima / limit).astype(np.float32)
    # Combine or mask array elements to form `q`.
    q = np.clip(np.rint(weights / scale), -limit, limit).astype(np.int8)
    return q, scale

# Allocate initialized array `(zero_codes, zero_scales)` with the specified shape and dtype.
zero_codes, zero_scales = quantize_weights(np.zeros((3,2),np.float32), 8)
# Allocate initialized array `` with the specified shape and dtype.
np.testing.assert_array_equal(zero_codes, np.zeros((3,2),np.int8))
# Allocate initialized array `` with the specified shape and dtype.
np.testing.assert_array_equal(zero_scales, np.ones((1,2),np.float32))

Code building blocks in this step

quantize_weights(...) — Call quantize_weights with your updated parameters or inputs from this lesson's workspace.
contract(...) — Call contract with your updated parameters or inputs from this lesson's workspace.

An all-zero channel uses scale $1$ and zero codes, so reconstruction stays zero without division by zero.

4 Check reconstructed outputs against a bound

Append this block to the same main.py and rerun the whole file. Reconstruct the weights, then compare observed output errors with the independent absolute-product bound.

main.py · append build step 4

# Step 4 — Check reconstructed outputs against a bound: Every error must fit its stated bound.
for bits in (8, 4):
    q, scale = quantize_weights(w, bits)
    reconstructed = q.astype(np.float32) * scale
    assert np.all(np.abs(reconstructed-w) <= scale / 2 + 1e-6)
    # Independent error bound: |x deltaW| <= |x| |deltaW|.
    error = np.abs(x @ reconstructed-reference)
    bound = np.abs(x) @ np.broadcast_to(scale / 2, w.shape)
    assert np.all(error <= bound + 1e-6)
    print("W%dA32 simulated output error" % bits, float(error.max()))

Code building blocks in this step

in(...) — Call in with your updated parameters or inputs from this lesson's workspace.
quantize_weights(...) — Call quantize_weights with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

Every error must fit its stated bound. Move on to activation clipping and the affine integer experiment before calling the policy W8A8.

View the cumulative build in main.py

These blocks append in order to the same file.

main.py · cumulative Python build

# Step 1 — Write the full-precision reference: The top-left output is -1.437.
import numpy as np
import jax.numpy as jnp
x = np.array([[.3, -1.2, 2.1], [1.1, .2, -.8]], np.float32)
w = np.array([[.11, -1.8], [.7, .2], [-.3, 2.4]], np.float32)
# Perform matrix / vector contraction (`@`) to compute `reference`.
reference = x @ w

np.testing.assert_allclose(reference[0,0], -1.437, atol=1e-6)

# Step 2 — Round inputs and weights, then accumulate explicitly: The printed errors isolate the consequence of representational...
def mixed_forward(x, w, dtype):
    # Demonstrate rounding inputs/weights with explicit float32 accumulation.
    a = jnp.asarray(x, dtype).astype(jnp.float32)
    # Create device-backed JAX array `b`.
    b = jnp.asarray(w, dtype).astype(jnp.float32)
    return np.asarray(a @ b)
for dtype in (jnp.float32, jnp.float16, jnp.bfloat16):
    y = mixed_forward(x, w, dtype)
    # Aggregate array values to compute `error`.
    error = float(np.max(np.abs(y-reference)))
    # Confirm that all computed values remain finite (no NaN or Inf).
    assert np.isfinite(y).all() and error < .05
    print(str(dtype), "max absolute error", error)

# Step 3 — Construct signed integer codes and channel scales: An all-zero channel uses scale 1 and zero codes, so reconstruction...
def quantize_weights(weights, bits):
    # Guard input contract (`bits not in (4, 8)`) and fail fast if violated.
    if bits not in (4, 8):
        raise ValueError("use signed 4 or 8 bit symmetric quantization")
    limit = 2 ** (bits - 1) - 1
    # Reduce along axis=0 to compute `maxima`.
    maxima = np.max(np.abs(weights), axis=0, keepdims=True)
    scale = np.where(maxima == 0, 1., maxima / limit).astype(np.float32)
    # Combine or mask array elements to form `q`.
    q = np.clip(np.rint(weights / scale), -limit, limit).astype(np.int8)
    return q, scale

# Allocate initialized array `(zero_codes, zero_scales)` with the specified shape and dtype.
zero_codes, zero_scales = quantize_weights(np.zeros((3,2),np.float32), 8)
# Allocate initialized array `` with the specified shape and dtype.
np.testing.assert_array_equal(zero_codes, np.zeros((3,2),np.int8))
# Allocate initialized array `` with the specified shape and dtype.
np.testing.assert_array_equal(zero_scales, np.ones((1,2),np.float32))

# Step 4 — Check reconstructed outputs against a bound: Every error must fit its stated bound.
for bits in (8, 4):
    q, scale = quantize_weights(w, bits)
    reconstructed = q.astype(np.float32) * scale
    assert np.all(np.abs(reconstructed-w) <= scale / 2 + 1e-6)
    # Independent error bound: |x deltaW| <= |x| |deltaW|.
    error = np.abs(x @ reconstructed-reference)
    bound = np.abs(x) @ np.broadcast_to(scale / 2, w.shape)
    assert np.all(error <= bound + 1e-6)
    print("W%dA32 simulated output error" % bits, float(error.max()))

Run the example

python3 phases/15-deployment/05-weight-activation-and-accumulation-precision/code/main.py

About this command

This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.

Inspect the complete assembled example

main.py · complete Python example

# Choose weight, activation and accumulation precision: A precision policy specifies how weights, activations,...
import numpy as np
import jax.numpy as jnp
x = np.array([[.3, -1.2, 2.1], [1.1, .2, -.8]], np.float32)
w = np.array([[.11, -1.8], [.7, .2], [-.3, 2.4]], np.float32)
# Perform matrix / vector contraction (`@`) to compute `reference`.
reference = x @ w
def mixed_forward(x, w, dtype):
    # Demonstrate rounding inputs/weights with explicit float32 accumulation.
    a = jnp.asarray(x, dtype).astype(jnp.float32)
    # Create device-backed JAX array `b`.
    b = jnp.asarray(w, dtype).astype(jnp.float32)
    return np.asarray(a @ b)
for dtype in (jnp.float32, jnp.float16, jnp.bfloat16):
    y = mixed_forward(x, w, dtype)
    # Aggregate array values to compute `error`.
    error = float(np.max(np.abs(y-reference)))
    # Confirm that all computed values remain finite (no NaN or Inf).
    assert np.isfinite(y).all() and error < .05
    print(str(dtype), "max absolute error", error)

def quantize_weights(weights, bits):
    # Guard input contract (`bits not in (4, 8)`) and fail fast if violated.
    if bits not in (4, 8):
        raise ValueError("use signed 4 or 8 bit symmetric quantization")
    limit = 2 ** (bits - 1) - 1
    # Reduce along axis=0 to compute `maxima`.
    maxima = np.max(np.abs(weights), axis=0, keepdims=True)
    scale = np.where(maxima == 0, 1., maxima / limit).astype(np.float32)
    # Combine or mask array elements to form `q`.
    q = np.clip(np.rint(weights / scale), -limit, limit).astype(np.int8)
    return q, scale
for bits in (8, 4):
    q, scale = quantize_weights(w, bits)
    reconstructed = q.astype(np.float32) * scale
    assert np.all(np.abs(reconstructed-w) <= scale / 2 + 1e-6)
    # Independent error bound: |x deltaW| <= |x| |deltaW|.
    error = np.abs(x @ reconstructed-reference)
    bound = np.abs(x) @ np.broadcast_to(scale / 2, w.shape)
    assert np.all(error <= bound + 1e-6)
    print("W%dA32 simulated output error" % bits, float(error.max()))

Expected result: FP32, FP16 and BF16 comparisons are finite with error below $0.05$ for this fixture. INT8/INT4 reconstruction errors obey a half-scale bound; these are simulated weight-only computations.

Recorded reference execution

Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.

2026-10-20T10:50:41.243511+00:00 · Python 3.14.3 · JAX 0.9.2 · 1 CPU device

<class 'jax.numpy.float32'> max absolute error 0.0
<class 'jax.numpy.float16'> max absolute error 0.0010547637939453125
<class 'jax.numpy.bfloat16'> max absolute error 0.006171703338623047
W8A32 simulated output error 0.008031368255615234
W4A32 simulated output error 0.14571428298950195
<class 'jax.numpy.float32'> max absolute error 0.0
<class 'jax.numpy.float16'> max absolute error 0.0010547637939453125
<class 'jax.numpy.bfloat16'> max absolute error 0.006171703338623047
W8A32 simulated output error 0.008031368255615234
W4A32 simulated output error 0.14571428298950195
Affine accumulators: [[5, 15]] output codes: [[2, 7]]
Correct / missing zero point / clipped: [[0.625, 0.9375]] [[-0.125, 0.5625]] [[0.52734375, 0.52734375]]
Clipped values: 1 reconstructed: [0.19685039 0.8976378  1.        ]
Affine accumulators: [[5, 15]] output codes: [[2, 7]]
Correct / missing zero point / clipped: [[0.625, 0.9375]] [[-0.125, 0.5625]] [[0.52734375, 0.52734375]]
Small-channel max error; shared / per-channel: 0.03 7.874053e-05
Zero channel handled
W8A8 fixture error 0.00742650032043457
Code-origin invariance and bias-only zero input verified.
Weight-plus-scale subtotals: 24 14 11
PASS: deployment-05

Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.

Calibrate an activation range, then shift it

Calibrate an activation range, then shift it · Python experiment

# Experiment — Calibrate an activation range, then shift it: Do not recalibrate on the test set to hide this error.
calibration = np.linspace(-1., 1., 101, dtype=np.float32)
# Aggregate array values to compute `activation_scale`.
activation_scale = float(np.max(np.abs(calibration))) / 127
held_out = np.array([.2, .9, 4.], np.float32)
# Combine or mask array elements to form `activation_codes`.
activation_codes = np.clip(np.rint(held_out / activation_scale), -127, 127).astype(np.int8)
restored_activations = activation_codes.astype(np.float32) * activation_scale
clipped = np.count_nonzero(np.abs(held_out) > 127 * activation_scale)
assert clipped == 1 and abs(float(restored_activations[-1])-4.) > 2.9
print("Clipped values:", clipped, "reconstructed:", restored_activations)

Expected: One value clips and reconstructs near $1$ instead of $4$ .

Do not recalibrate on the test set to hide this error. Collect representative calibration data or choose a policy that meets the task requirements.

Include zero points, bias and the output grid

Include zero points, bias and the output grid · Python experiment

# Deliberately exact grid values make an independent hand calculation possible.
a_sx, a_zx = .25, -3
a_sw = np.array([.5, .25], np.float64)
a_qx = np.array([[-3, 1, 5]], np.int8)
a_qw = np.array([[2, -1], [-1, 2], [1, 1]], np.int8)
a_bias = np.array([.125, -.0625], np.float64)
a_bias_codes = np.rint(a_bias / (a_sx * a_sw)).astype(np.int64)
# Check the wide result before representing it with an INT32 accumulator.
a_wide = (a_qx.astype(np.int64) - a_zx) @ a_qw.astype(np.int64) + a_bias_codes
assert np.all((a_wide >= np.iinfo(np.int32).min) & (a_wide <= np.iinfo(np.int32).max))
a_acc = a_wide.astype(np.int32)
# Execute `np.testing.assert_array_equal(a_acc, [[5, 15]])`
np.testing.assert_array_equal(a_acc, [[5, 15]])
a_real = a_acc.astype(np.float64) * a_sx * a_sw
# Independent real-valued input/weights, written without decoding the codes.
a_reference = np.array([[0., 1., 2.]]) @ np.array([[1., -.25], [-.5, .5], [.5, .25]]) + a_bias
# Execute `np.testing.assert_array_equal(a_real, [[.625, .9375]])`
np.testing.assert_array_equal(a_real, [[.625, .9375]])
np.testing.assert_allclose(a_real, a_reference, atol=1e-12, rtol=0)
def output_grid(real, scale, zero_point):
    # Guard input contract (`not np.isfinite(scale) or scale <= 0`) and fail fast if violated.
    if not np.isfinite(scale) or scale <= 0:
        raise ValueError('positive finite output scale required')
    # Guard input contract (`type(zero_point) is not int or not -128 <= zero_point <= 127`) and fail fast if violated.
    if type(zero_point) is not int or not -128 <= zero_point <= 127:
        raise ValueError('integer INT8 zero point required')
    unbounded = np.rint(real / scale) + zero_point
    clipped = (unbounded < -128) | (unbounded > 127)
    # Combine or mask array elements to form `codes`.
    codes = np.clip(unbounded, -128, 127).astype(np.int8)
    return codes, (codes.astype(np.float64) - zero_point) * scale, clipped

a_codes, a_restored, a_clipped = output_grid(a_real, .0625, -8)
# Execute `np.testing.assert_array_equal(a_codes, [[2, 7]])`
np.testing.assert_array_equal(a_codes, [[2, 7]])
assert not a_clipped.any()
np.testing.assert_allclose(a_restored, a_reference, atol=1e-12, rtol=0)
a_wrong = (a_qx.astype(np.int64) @ a_qw.astype(np.int64) + a_bias_codes) * a_sx * a_sw
# Execute `np.testing.assert_array_equal(a_wrong, [[-.125, .5625]])`
np.testing.assert_array_equal(a_wrong, [[-.125, .5625]])
_, a_saturated, a_small_clipped = output_grid(a_real, 1./256, -8)
assert a_small_clipped.all()
# Execute `np.testing.assert_array_equal(a_saturated, [[135./256, 135./`
np.testing.assert_array_equal(a_saturated, [[135./256, 135./256]])
print('Affine accumulators:', a_acc.tolist(), 'output codes:', a_codes.tolist())
print('Correct / missing zero point / clipped:', a_restored.tolist(), a_wrong.tolist(), a_saturated.tolist())

Expected: Accumulators [[5, 15]], output codes [[2, 7]], correct outputs [[0.625, 0.9375]]. Omitting the input zero point gives [[-0.125, 0.5625]]. The narrow output grid clips both outputs to [[0.52734375, 0.52734375]].

The first grid represents both values exactly. With scale $1/256$ , the highest output is $(127+8)/256=0.52734375$ , so both values saturate. A half-step rounding bound applies only without clipping. This is a NumPy arithmetic audit, not a native INT8 kernel or bit-exact LiteRT implementation: real runtimes specify operator layouts, multiplier approximations and rounding rules separately.

Give small and large output channels different grids

Give small and large output channels different grids · Python experiment

# Experiment — Give small and large output channels different grids: The global maximum determines a coarse grid for every channel.
heterogeneous = np.array([[.01, 10.], [-.02, -5.], [.03, 2.]], np.float32)
channel_codes, channel_scales = quantize_weights(heterogeneous, 8)
per_channel = channel_codes.astype(np.float32) * channel_scales
# Aggregate array values to compute `shared_scale`.
shared_scale = np.max(np.abs(heterogeneous)) / 127
shared = np.clip(np.rint(heterogeneous/shared_scale), -127, 127).astype(np.int8).astype(np.float32)*shared_scale
assert np.all(shared[:,0] == 0)
assert np.max(np.abs(per_channel[:,0]-heterogeneous[:,0])) < .001
np.testing.assert_allclose(shared[:,0], [0.,0.,0.], atol=0)
print('Small-channel max error; shared / per-channel:', np.max(np.abs(shared[:,0]-heterogeneous[:,0])), np.max(np.abs(per_channel[:,0]-heterogeneous[:,0])))

Expected: The shared scale rounds the entire small channel to zero. Its error is $0.03$ ; the per-channel error stays below $0.001$ .

The global maximum determines a coarse grid for every channel. A per-channel grid protects the small channel here, at the cost of additional scale metadata and a runtime contract that must support the chosen axis.

Hands-On Exercise · Modify & Verify Run in the declared lesson environment

Make it yours

Where to put your code: Append your solution at the bottom of phases/15-deployment/05-weight-activation-and-accumulation-precision/code/main.py (or inside the Make it yours cell of exercises/deployment-05.ipynb) and run python3 phases/15-deployment/05-weight-activation-and-accumulation-precision/code/main.py to verify your prediction before opening the reference solution.

Add a zero-valued output channel and verify that its quantization scale is finite and its reconstruction is exactly zero.

Show a writing guide and starter scaffold

How to write this exercise code: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

jnp.isfinite(x) — Returns a boolean mask verifying that no element is NaN or Inf.

2. Step-by-step implementation plan:

Confirm that all computed values remain finite (no NaN or Inf).
Execute np.testing.assert_array_equal(q[:, -1], 0)
Execute np.testing.assert_array_equal((q * scale)[:, -1], 0)

Starter code scaffold · copy into your file & fill in the TODOs

# Exercise solution: Add a zero-valued output channel and verify that its quantization...
with_zero = np.column_stack(...)  # TODO: compute with_zero
q, scale = quantize_weights(...)  # TODO: compute q, scale
# Confirm that all computed values remain finite (no NaN or Inf).
assert np.isfinite(scale).all()  # TODO: complete assertion check
# Execute `np.testing.assert_array_equal(q[:, -1], 0)`
np.testing.assert_array_equal(q[:, -1], 0)
# Execute `np.testing.assert_array_equal((q * scale)[:, -1], 0)`
np.testing.assert_array_equal((q * scale)[:, -1], 0)
print("Zero channel handled")

Reveal the reference solution

Reference solution

# Exercise solution: Add a zero-valued output channel and verify that its quantization...
with_zero = np.column_stack((w, np.zeros(3, np.float32)))
q, scale = quantize_weights(with_zero, 8)
# Confirm that all computed values remain finite (no NaN or Inf).
assert np.isfinite(scale).all()
# Execute `np.testing.assert_array_equal(q[:, -1], 0)`
np.testing.assert_array_equal(q[:, -1], 0)
# Execute `np.testing.assert_array_equal((q * scale)[:, -1], 0)`
np.testing.assert_array_equal((q * scale)[:, -1], 0)
print("Zero channel handled")

Compute W8A8 with a wide accumulator

Transfer

Use calibrated symmetric scales for $x$ and per-channel scales for $w$ . Multiply integer codes with INT32 accumulation, rescale, and compare to the float baseline. Do not multiply int8 arrays directly.

Show a writing guide and starter scaffold

How to write this practice solution: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

jnp.allclose(actual, expected, rtol=..., atol=...) — Checks that two arrays match elementwise within floating-point tolerance.

2. Step-by-step implementation plan:

Aggregate array values to compute sx.
Combine or mask array elements to form qx.
Perform matrix / vector contraction (@) to compute accumulator.

Starter code scaffold · copy into your file & fill in the TODOs

# Compute W8A8 with a wide accumulator (Transfer): This dense fixture uses zero points of zero and no bias.
# Aggregate array values to compute `sx`.
sx = float(...)  # TODO: compute sx
# Combine or mask array elements to form `qx`.
qx = np.clip(...)  # TODO: compute qx
qw, sw = quantize_weights(...)  # TODO: compute qw, sw
# Perform matrix / vector contraction (`@`) to compute `accumulator`.
accumulator = qx.astype(...)  # TODO: compute accumulator
y_integer = accumulator.astype(...)  # TODO: compute y_integer
assert accumulator.dtype  # TODO: complete assertion check
np.testing.assert_allclose(y_integer, reference, atol = ...  # TODO: compute np.testing.assert_allclose(y_integer, reference, atol
print("W8A8 fixture error", float(np.max(np.abs(y_integer-reference))))

Show a hint

Cast both integer-code arrays before multiplication. The real accumulator unit is the activation scale multiplied by each output channel’s weight scale; compare with the unchanged floating-point reference.

Reveal reference solution and reasoning

Compute W8A8 with a wide accumulator · Python solution

# Compute W8A8 with a wide accumulator (Transfer): This dense fixture uses zero points of zero and no bias.
# Aggregate array values to compute `sx`.
sx = float(np.max(np.abs(x))) / 127
# Combine or mask array elements to form `qx`.
qx = np.clip(np.rint(x / sx), -127, 127).astype(np.int8)
qw, sw = quantize_weights(w, 8)
# Perform matrix / vector contraction (`@`) to compute `accumulator`.
accumulator = qx.astype(np.int32) @ qw.astype(np.int32)
y_integer = accumulator.astype(np.float32) * sx * sw
assert accumulator.dtype == np.int32
np.testing.assert_allclose(y_integer, reference, atol=.06, rtol=0)
print("W8A8 fixture error", float(np.max(np.abs(y_integer-reference))))

This dense fixture uses zero points of zero and no bias. A real affine operator must subtract zero points and handle bias and output requantization. Wide accumulation avoids int8 overflow; sufficiently long reductions can still overflow INT32.

Change the code origin without changing the values

Transfer

Change the activation zero point from $-3$ to $17$ , shift the input codes accordingly, and prove that predictions stay unchanged. Then feed codes representing real zero and identify the expected bias-only result.

Show a writing guide and starter scaffold

How to write this practice solution: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

values(...) — Call values with your updated parameters or inputs from this lesson's workspace.
a_qx.astype(...) — Call a_qx.astype with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

2. Step-by-step implementation plan:

Execute np.testing.assert_array_equal(shifted_acc, a_acc)
Execute np.testing.assert_array_equal(zero_result, a_bias[None, :])

Starter code scaffold · copy into your file & fill in the TODOs

# Change the code origin without changing the values (Transfer): Integer codes are coordinates on a grid.
shifted_zx = ...  # TODO: compute shifted_zx
shifted_codes_wide = a_qx.astype(...)  # TODO: compute shifted_codes_wide
assert np.all((-128  # TODO: complete assertion check
shifted_codes = shifted_codes_wide.astype(...)  # TODO: compute shifted_codes
shifted_acc = ...  # TODO: compute shifted_acc
# Execute `np.testing.assert_array_equal(shifted_acc, a_acc)`
np.testing.assert_array_equal(shifted_acc, a_acc)
zero_codes = np.full(...)  # TODO: compute zero_codes
zero_result = ...  # TODO: compute zero_result
# Execute `np.testing.assert_array_equal(zero_result, a_bias[None, :])`
np.testing.assert_array_equal(zero_result, a_bias[None, :])
for bad_scale in (0., -1., float('nan')):
    try:
        output_grid(a_real, bad_scale, -8)
    except ValueError:
        pass
    else:
        raise AssertionError('invalid scale accepted')
print('Code-origin invariance and bias-only zero input verified.')

Show a hint

Shift the zero point and codes by the same amount before subtracting; verify representability before casting back to INT8.

Reveal reference solution and reasoning

Change the code origin without changing the values · Python solution

# Change the code origin without changing the values (Transfer): Integer codes are coordinates on a grid.
shifted_zx = 17
shifted_codes_wide = a_qx.astype(np.int64) + (shifted_zx - a_zx)
assert np.all((-128 <= shifted_codes_wide) & (shifted_codes_wide <= 127))
shifted_codes = shifted_codes_wide.astype(np.int8)
shifted_acc = (shifted_codes.astype(np.int64) - shifted_zx) @ a_qw.astype(np.int64) + a_bias_codes
# Execute `np.testing.assert_array_equal(shifted_acc, a_acc)`
np.testing.assert_array_equal(shifted_acc, a_acc)
zero_codes = np.full((1, 3), shifted_zx, np.int8)
zero_result = ((zero_codes.astype(np.int64) - shifted_zx) @ a_qw.astype(np.int64) + a_bias_codes) * a_sx * a_sw
# Execute `np.testing.assert_array_equal(zero_result, a_bias[None, :])`
np.testing.assert_array_equal(zero_result, a_bias[None, :])
for bad_scale in (0., -1., float('nan')):
    try:
        output_grid(a_real, bad_scale, -8)
    except ValueError:
        pass
    else:
        raise AssertionError('invalid scale accepted')
print('Code-origin invariance and bias-only zero input verified.')

Integer codes are coordinates on a grid. Changing the coordinate origin consistently preserves represented values. The zero-input check exposes a missing zero-point correction or bias in the wrong accumulator units.

Account for actual low-bit storage

Transfer / diagnosis

Calculate FP32 storage, ideal packed four-bit storage, and the actual int8-backed four-bit simulation for the fixture. Include per-output-channel FP32 scales; keep biases and runtime workspace explicitly outside this subtotal.

Show a writing guide and starter scaffold

How to write this practice solution: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

storage(...) — Call storage with your updated parameters or inputs from this lesson's workspace.
quantize_weights(...) — Call quantize_weights with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

2. Step-by-step implementation plan:

Set up the changed input or configuration parameters requested in the prompt.
Run the target function or transformation and assert that the output matches your prediction.

Starter code scaffold · copy into your file & fill in the TODOs

# Account for actual low-bit storage (Transfer / diagnosis): The simulation uses 14 bytes for its codes and scales, not...
q4, s4 = quantize_weights(...)  # TODO: compute q4, s4
float_weight_bytes = ...  # TODO: compute float_weight_bytes
simulated_weight_and_scale_bytes = ...  # TODO: compute simulated_weight_and_scale_bytes
ideal_packed_weight_and_scale_bytes = ...  # TODO: compute ideal_packed_weight_and_scale_bytes
assert (float_weight_bytes, simulated_weight_and_scale_bytes, ideal_packed_weight_and_scale_bytes)  # TODO: complete assertion check
print('Weight-plus-scale subtotals:', float_weight_bytes, simulated_weight_and_scale_bytes, ideal_packed_weight_and_scale_bytes)

Show a hint

Use nbytes for actual arrays. Packing two signed four-bit codes into one byte is not implemented by astype(int8).

Reveal reference solution and reasoning

Account for actual low-bit storage · Python solution

# Account for actual low-bit storage (Transfer / diagnosis): The simulation uses 14 bytes for its codes and scales, not...
q4, s4 = quantize_weights(w, 4)
float_weight_bytes = w.nbytes
simulated_weight_and_scale_bytes = q4.nbytes + s4.nbytes
ideal_packed_weight_and_scale_bytes = (w.size + 1)//2 + s4.nbytes
assert (float_weight_bytes, simulated_weight_and_scale_bytes, ideal_packed_weight_and_scale_bytes) == (24,14,11)
print('Weight-plus-scale subtotals:', float_weight_bytes, simulated_weight_and_scale_bytes, ideal_packed_weight_and_scale_bytes)

The simulation uses $14$ bytes for its codes and scales, not the ideally packed $11$ . Tiny tensors emphasize metadata overhead; large models also need block/group metadata, alignment, biases and working memory.

Diagnose the result

NaNs in FP16 suggest checking range and sensitive reductions. Finite but degraded predictions suggest rounding or clipping; inspect tensor ranges and per-channel errors. No memory improvement from the INT4 exercise is expected because its codes use int8 storage. No latency gain after conversion calls for profiling fallback operators and data transfers on the target.

Keep your evidence

Download evidence template

Keep a precision table listing weight, activation, accumulator and output dtypes; max error and clipping counts on changed inputs; calibration provenance; and storage estimates that include scales and packing.

Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.

Saved in this browser. Export or back up your portfolio anytime in My learning.

Save work for this lesson

Verified on local CPU. Stage to a Cloud TPU VM via Cloud TPU setup to record accelerator execution receipts (TODO: add automated Cloud TPU CI execution receipts).

Choose weight, activation and accumulation precision

What you will be able to do

The problem

The idea

Follow the units through integer arithmetic

Integer values need scales and zero-points

Write the policy before choosing a dtype

Mixed precision during training

Quantize weights with scale metadata

Weight-only and full integer are different deployments

FP8 and other low-bit formats require a target contract

Evaluate quality separately from size and speed

Follow one value onto the quantization grid

Follow an affine output through every numeric unit

Distinguish storage savings from an error budget

Follow rounding error through the next multiplication

Precision choices trade representation for output error

Read the figure

Connect it to the computation

1 Write the full-precision reference

2 Round inputs and weights, then accumulate explicitly

3 Construct signed integer codes and channel scales

4 Check reconstructed outputs against a bound

Run the example

Recorded reference execution

Calibrate an activation range, then shift it

Include zero points, bias and the output grid

Give small and large output channels different grids

Make it yours

Compute W8A8 with a wide accumulator

Change the code origin without changing the values

Account for actual low-bit storage

Check your understanding

Diagnose the result

Keep your evidence

Primary references