Phase 18 / Post-training: SFT, LoRA, reward models and RLHF
Learn a reward model from pairwise preferences
Revision v0.18.1 · Updated since your last visit. Review the changes; your progress is saved.
Lesson · about 105 minutes · CPU practice
Fit pairwise reward differences and evaluate unchanged held-out unanimous and disagreement cases against a pre-fit baseline.
Phase study guide: preparation, milestones and project evidence
Before you start
Before this: LoRA: adapt, save and merge low-rank updates
Hardware: CPU
Setup help: Python, notebook, or Cloud TPU VM
Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions require separate checks. Hardware receipts apply only to their recorded source, environment, hardware, and cases; they do not qualify every lesson or TPU configuration. See the TPU performance and hardware evidence guide. The optional Cloud TPU specialization explains target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.
Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txtpy -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txtAfter activation, run the complete worked example from that same folder:
python phases/18-posttraining/03-reward/code/main.py.\.venv\Scripts\python.exe phases/18-posttraining/03-reward/code/main.pyFor an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.
Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.
For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:
python -m pip install jupyterlab
python -m jupyter lab.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter labSelect the downloaded posttraining-03.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.
The worked reference for this lesson is CPU-validated. Some references explicitly configure logical CPU devices; setting JAX_PLATFORMS does not adapt them to a TPU. Follow the execution matrix and reviewed multi-host example before attempting a target run.
For accelerator preparation, study TPU setup, observed runtime receipts, checkpointed jobs and precision and retained traces. Current receipts cover bounded v5e1 launch, resume, and precision checks; four single-device Pallas cases; and a four-device v5e collective fixture. Each receipt is limited to its recorded source, environment, cases, and hardware. It does not qualify all lessons or TPU configurations. Cloud provisioning is optional for CPU learning.
Check: The model learns all three declared preferences; common reward offsets preserve probabilities.
Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.
What you will be able to do
- Represent a preference as a comparison
- Fit score differences with a stable likelihood
- Verify the changed-condition exercise and explain the limits of this fixture.
The problem
You know which of two responses is preferred, but you do not have an absolute score for either. How can a model learn from that comparison, and what does its score mean?
The idea
A pairwise reward model learns to give a preferred item a higher score than its alternative. The loss uses a score difference. That relative objective does not automatically calibrate absolute quality or make preference labels infallible.
A preference compares scores rather than calibrating utility
A preference label says which of two responses was favored under a shared prompt and rubric. The model turns the score difference into a preference probability: . That probability describes the label likelihood; each response still receives a scalar reward score, not a calibrated probability of being good. Adding the same constant to both rewards leaves unchanged.
In the recorded three-pair synthetic fit, mean pairwise NLL falls from 0.693 nats (a 50/50 tie) to 0.087, and all three training margins become positive. This shows the fitted ranking on those pairs, not human agreement. Before using the scores downstream, inspect held-out pairs and ask whether disagreements and unseen prompts are represented.
Before coding: a preference is a comparison
A training example says that one response was preferred to another for the same prompt and rubric. It does not assign a universally meaningful numeric score to either response. Begin by subtracting the rejected score from the chosen score. A positive margin supports the label; a negative margin contradicts it.
The fixture uses three synthetic response feature vectors and three ordered pairs. It teaches comparison likelihood and score identifiability, not human judgment. Keep the prompt/group identity when replacing these vectors with real response representations; comparing unrelated prompts can teach a meaningless ranking.
Represent a preference as a comparison
Keep the prompt identity, chosen/rejected response identities, label source and split. Pairwise ratings are conditional on a shared prompt and rubric; comparing unrelated prompts can teach a spurious score offset. The fixture has three response feature vectors with an explicit ordering.
Fit score differences with a stable likelihood
For margin , the chance assigned to the observed preference is , while the training loss is . At , the probability is one half and loss is ; at , the probability is three quarters. Those are probabilities of the pairwise label under the model, not probabilities that a reward score is correct in general. The scalar score itself remains relative: the shared prompt comparison cancels any common offset.
Calculate likelihood and its derivative
Let . The probability of the observed preference is , and its negative log likelihood is . At margin zero, probability is one half and loss is . At margin , probability is three quarters and loss is . The margin derivative is : confident correct pairs receive a smaller update than uncertain or wrong pairs.
For a linear score, multiply this derivative by the chosen-minus-rejected feature vector and average pairs. That supplies an independent gradient oracle without differentiating the library softplus implementation.
Disagreement changes the optimum
Conflicting labels provide evidence about uncertainty. If the same pair appears once in each direction, the average likelihood is best at margin zero, which assigns one-half to each direction; forcing a huge positive margin would fit one label and contradict the other. For uneven votes, the fitted probability can reflect their frequency. Preserve the vote fraction and source rather than reducing every pair to an unquestioned majority.
Separate ranking from calibration
On the held-out synthetic set, the pre-fit zero-score model assigns 0.5 to all four comparisons, with mean Brier error 0.1406 and 0/1 accuracy on unanimous cases. After fitting the three training pairs, held-out Brier error is 0.0223 and unanimous accuracy is 1/1. On the two disagreement cases, Brier error changes only from 0.0313 to 0.0294; their probabilities are retained rather than forced into a unanimous label. This is a tiny synthetic check, not human or language-response validation. Use held-out slices to decide whether this reward is ready for the next experiment.
Preserve disagreement and data provenance
Real raters may disagree for legitimate reasons. Retain the rating rubric, task, annotator process and uncertainty instead of converting every disagreement into a perfect label. Split prompts and near-duplicate responses by source so a held-out score does not reward memorization. This lab does not collect personal data or human ratings.
Keep the reward model fixed during policy evaluation
The next lesson freezes this reward model while a policy changes. If scores moved during that comparison, a policy could appear to improve because the evaluator changed. Carry the reward-model identity and its held-out behavior with the run, keep an independent task metric, and inspect outputs the reward may score incorrectly. A rising reward alone is not evidence of better responses.
Audit both what scores identify and what they do not
Pairwise comparisons identify relative margins, not an absolute zero: adding a common offset changes no training probability. Scaling the weights preserves ordering but changes confidence and the reward scale, which changes how strongly a downstream KL penalty competes with reward. In this recorded fit, the margins are about 1.65, 4.29, and 2.64 for the three ordered pairs. Save model identity and normalization so the next policy run uses a known score scale.
Freeze a baseline before fitting and inspect held-out disagreement cases
The three comparisons in the fitted example define training only. Add a separate fixed set of held-out feature pairs, including unanimous and split votes, before fitting the reward weights. Compute baseline pairwise probabilities from zero weights first, then score the same cases after the update. Report held-out Brier error and preference accuracy for unanimous cases; for disagreement cases, report predicted probabilities and Brier error against the vote fraction rather than forcing a majority label.
This small finite fixture can expose a broken pairwise objective or a train/held-out mismatch. Its scores and votes are synthetic. They are not human reward calibration, language-response usefulness, or evidence that a downstream policy should optimize these scores.
Learn a reward model from pairwise preferences — recorded experiment
Predict before looking: Which final comparison margin should be the largest given the ordering zero above one above two?
Read the figure
The first panel uses completed parameter updates on x and pairwise NLL in nats on y; on the same three synthetic training pairs it falls from 0.693 (a tied 50/50 prediction) to 0.087. The second panel gives final chosen-minus-rejected reward margins: about 1.65, 4.29, and 2.64 for 0>1, 0>2, and 1>2. All are positive, and the widest comparison has the largest margin.
Connect it to the computation
This confirms the model learned the fixture's three relative orderings, not an absolute or universally calibrated notion of quality. The held-out report is a separate check: Brier error falls from 0.1406 to 0.0223 and unanimous accuracy rises from 0/1 to 1/1, while disagreement Brier changes only from 0.0313 to 0.0294. Because the held-out set has only four synthetic comparisons, use a larger prompt-grouped evaluation before trusting the score in policy optimization.
Figure data and execution details
Open full-size figureDownload data and provenance
The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.
Your build: Create a separate main.py in your practice folder. Append each block in order and run python main.py in the lesson environment. The downloaded script is the complete reference, so keep it separate from your attempt.
1 Define chosen-minus-rejected likelihood
Create main.py in your activated course environment. Paste this block, then run python main.py; function definitions alone print nothing.
# Step 1 — 1. Define chosen-minus-rejected likelihood: The loss sees feature differences.
import jax
import jax.numpy as jnp
import numpy as np
def preference_loss(w, chosen, rejected):
margin = (chosen - rejected) @ w
return jnp.mean(jax.nn.softplus(-margin))The loss sees feature differences. Predict why adding a common score offset later cannot change it.
Explain the APIs in this step
jnp.mean(x, axis=..., keepdims=...): Averages JAX array values along the selected axis.
2 Construct and inspect the three comparisons
Append this block to main.py and run python main.py again. Keep the earlier blocks above it.
# Step 2 — 2. Construct and inspect the three comparisons
features = jnp.array([[1.0, 0.0], [0.0, 1.0], [-1.0, -1.0]])
chosen = features[jnp.array([0, 0, 1])]
rejected = features[jnp.array([1, 2, 2])]
w = jnp.zeros(2)
history = []
step = jax.jit(
jax.value_and_grad(lambda w: preference_loss(w, chosen, rejected))
)Write the desired order before fitting: response zero above one, and one above two. These labels are synthetic.
Explain the APIs in this step
jnp.array(values, dtype=...): Constructs a JAX array from Python/NumPy values.jnp.zeros(shape, dtype=...): Creates a JAX array of the requested shape filled with zeros.jax.jit(fn) / @jax.jit: Traces a function for its input signatures and compiles it; changed shapes, dtypes, or static arguments can trigger another compilation.jax.value_and_grad(loss_fn)(params, ...): Transforms a pure function to return its value and gradient(value, grads)from the same evaluation.
3 Freeze held-out reward pairs before fitting
Insert this block after constructing the three training comparisons and before the fit loop. It records the fixed synthetic held-out baseline before reward weights are updated.
# Held-out examples are fixed before the optimizer loop; they are not used for fitting.
heldout_chosen = jnp.array([[1.0, 0.0], [0.0, 1.0], [0.2, 0.2], [-1.0, -1.0]])
heldout_rejected = jnp.array([[0.0, 1.0], [-1.0, -1.0], [-0.1, 0.0], [0.0, 1.0]])
heldout_votes = np.array([1.0, 1.0, 0.5, 0.25], dtype=np.float64)
heldout_slices = np.array(["unanimous", "unanimous", "disagreement", "disagreement"])
w_baseline = jnp.zeros_like(w)
def heldout_report(weights):
margins = np.asarray((heldout_chosen - heldout_rejected) @ weights, dtype=np.float64)
probabilities = 1.0 / (1.0 + np.exp(-margins))
brier = (probabilities - heldout_votes) ** 2
unanimous = heldout_slices == "unanimous"
disagreement = ~unanimous
return {"probabilities": probabilities, "brier_mean": float(np.mean(brier)),
"unanimous_accuracy": float(np.mean((probabilities[unanimous] > 0.5) == (heldout_votes[unanimous] > 0.5))),
"disagreement_probabilities": probabilities[disagreement],
"disagreement_brier": float(np.mean(brier[disagreement]))}
pretraining_heldout_baseline = heldout_report(w)
assert np.allclose(pretraining_heldout_baseline["probabilities"], 0.5)
print("Frozen held-out reward baseline before fit:", pretraining_heldout_baseline)This code runs after the training comparisons are defined and before the optimizer loop. The zero-weight probabilities and fractional-vote Brier score are the baseline for this fixed held-out set.
Explain the APIs in this step
jnp.array(values, dtype=...): Constructs a JAX array from Python/NumPy values.np.allclose(actual, expected, rtol=..., atol=...): Returns a Python boolean indicating whether NumPy-convertible values satisfy the tolerance test.
4 Fit rewards and check their invariances
Append this block to main.py and run python main.py again. Keep the earlier blocks above it.
# Step 3 — 3. Fit rewards and check their invariances: The host logaddexp oracle checks stable likelihood values.
for _ in range(100):
value, g = step(w)
history.append(float(value))
w = w - 0.1 * g
rewards = features @ w
assert rewards[0] > rewards[1] > rewards[2]
np.testing.assert_allclose(history[0], np.log(2), atol=1e-6)
assert history[-1] < 0.1
# Transfer the device result to the host for an independent NumPy comparison.
margins = np.asarray((chosen - rejected) @ w, dtype=np.float64)
np.testing.assert_allclose(
preference_loss(w, chosen, rejected),
np.mean(np.logaddexp(0, -margins)),
atol=1e-6,
)
# Only differences are identified: a common score offset leaves preference probabilities unchanged.
np.testing.assert_allclose(
jax.nn.sigmoid((chosen @ w + 9) - (rejected @ w + 9)),
jax.nn.sigmoid((chosen - rejected) @ w),
atol=1e-6,
)
print(
'Synthetic preference NLL initial/final:',
history[0],
history[-1],
'; rewards:',
np.asarray(rewards),
)The host logaddexp oracle checks stable likelihood values. The score-offset test establishes non-identifiability, not calibration.
Explain the APIs in this step
np.testing.assert_allclose(actual, desired, rtol=..., atol=...): Checks a host-side numerical expectation and raisesAssertionErrorwhen values differ beyond tolerance.
View the cumulative build in main.py
These blocks append in order to your separate practice file.
# Step 1 — 1. Define chosen-minus-rejected likelihood: The loss sees feature differences.
import jax
import jax.numpy as jnp
import numpy as np
def preference_loss(w, chosen, rejected):
margin = (chosen - rejected) @ w
return jnp.mean(jax.nn.softplus(-margin))
# Step 2 — 2. Construct and inspect the three comparisons
features = jnp.array([[1.0, 0.0], [0.0, 1.0], [-1.0, -1.0]])
chosen = features[jnp.array([0, 0, 1])]
rejected = features[jnp.array([1, 2, 2])]
w = jnp.zeros(2)
history = []
step = jax.jit(
jax.value_and_grad(lambda w: preference_loss(w, chosen, rejected))
)
# Held-out examples are fixed before the optimizer loop; they are not used for fitting.
heldout_chosen = jnp.array([[1.0, 0.0], [0.0, 1.0], [0.2, 0.2], [-1.0, -1.0]])
heldout_rejected = jnp.array([[0.0, 1.0], [-1.0, -1.0], [-0.1, 0.0], [0.0, 1.0]])
heldout_votes = np.array([1.0, 1.0, 0.5, 0.25], dtype=np.float64)
heldout_slices = np.array(["unanimous", "unanimous", "disagreement", "disagreement"])
w_baseline = jnp.zeros_like(w)
def heldout_report(weights):
margins = np.asarray((heldout_chosen - heldout_rejected) @ weights, dtype=np.float64)
probabilities = 1.0 / (1.0 + np.exp(-margins))
brier = (probabilities - heldout_votes) ** 2
unanimous = heldout_slices == "unanimous"
disagreement = ~unanimous
return {"probabilities": probabilities, "brier_mean": float(np.mean(brier)),
"unanimous_accuracy": float(np.mean((probabilities[unanimous] > 0.5) == (heldout_votes[unanimous] > 0.5))),
"disagreement_probabilities": probabilities[disagreement],
"disagreement_brier": float(np.mean(brier[disagreement]))}
pretraining_heldout_baseline = heldout_report(w)
assert np.allclose(pretraining_heldout_baseline["probabilities"], 0.5)
print("Frozen held-out reward baseline before fit:", pretraining_heldout_baseline)
# Step 3 — 3. Fit rewards and check their invariances: The host logaddexp oracle checks stable likelihood values.
for _ in range(100):
value, g = step(w)
history.append(float(value))
w = w - 0.1 * g
rewards = features @ w
assert rewards[0] > rewards[1] > rewards[2]
np.testing.assert_allclose(history[0], np.log(2), atol=1e-6)
assert history[-1] < 0.1
# Transfer the device result to the host for an independent NumPy comparison.
margins = np.asarray((chosen - rejected) @ w, dtype=np.float64)
np.testing.assert_allclose(
preference_loss(w, chosen, rejected),
np.mean(np.logaddexp(0, -margins)),
atol=1e-6,
)
# Only differences are identified: a common score offset leaves preference probabilities unchanged.
np.testing.assert_allclose(
jax.nn.sigmoid((chosen @ w + 9) - (rejected @ w + 9)),
jax.nn.sigmoid((chosen - rejected) @ w),
atol=1e-6,
)
print(
'Synthetic preference NLL initial/final:',
history[0],
history[-1],
'; rewards:',
np.asarray(rewards),
)Run the example
python3 phases/18-posttraining/03-reward/code/main.pyAbout this command
This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.
Inspect the complete assembled example
# Step 1 — 1. Define chosen-minus-rejected likelihood: The loss sees feature differences.
import jax
import jax.numpy as jnp
import numpy as np
def preference_loss(w, chosen, rejected):
margin = (chosen - rejected) @ w
return jnp.mean(jax.nn.softplus(-margin))
# Step 2 — 2. Construct and inspect the three comparisons
features = jnp.array([[1.0, 0.0], [0.0, 1.0], [-1.0, -1.0]])
chosen = features[jnp.array([0, 0, 1])]
rejected = features[jnp.array([1, 2, 2])]
w = jnp.zeros(2)
history = []
step = jax.jit(
jax.value_and_grad(lambda w: preference_loss(w, chosen, rejected))
)
# Step 3 — 3. Fit rewards and check their invariances: The host logaddexp oracle checks stable likelihood values.
for _ in range(100):
value, g = step(w)
history.append(float(value))
w = w - 0.1 * g
rewards = features @ w
assert rewards[0] > rewards[1] > rewards[2]
np.testing.assert_allclose(history[0], np.log(2), atol=1e-6)
assert history[-1] < 0.1
# Transfer the device result to the host for an independent NumPy comparison.
margins = np.asarray((chosen - rejected) @ w, dtype=np.float64)
np.testing.assert_allclose(
preference_loss(w, chosen, rejected),
np.mean(np.logaddexp(0, -margins)),
atol=1e-6,
)
# Only differences are identified: a common score offset leaves preference probabilities unchanged.
np.testing.assert_allclose(
jax.nn.sigmoid((chosen @ w + 9) - (rejected @ w + 9)),
jax.nn.sigmoid((chosen - rejected) @ w),
atol=1e-6,
)
print(
'Synthetic preference NLL initial/final:',
history[0],
history[-1],
'; rewards:',
np.asarray(rewards),
)Expected result: The model learns all three declared preferences; common reward offsets preserve probabilities.
Recorded output and runtime details
Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.
Frozen held-out reward baseline before fit: {'probabilities': array([0.5, 0.5, 0.5, 0.5]), 'brier_mean': 0.140625, 'unanimous_accuracy': 0.0, 'disagreement_probabilities': array([0.5, 0.5]), 'disagreement_brier': 0.03125}
Synthetic preference NLL initial/final: 0.6931471824645996 0.08669456839561462 ; rewards: [ 1.9811294 0.327278 -2.3084073]
Synthetic preference NLL initial/final: 0.6931471824645996 0.08669456839561462 ; rewards: [ 1.9811294 0.327278 -2.3084073]
Forward/reversed preference NLL: 0.08596369624137878 2.94565486907959
Balanced conflicting labels prefer a zero margin.
Tied baseline and preference directions verified.
Ranking unchanged; confidence and downstream reward scale change.
Independent reward gradients agree in both label directions.
heldout reward baseline: {'probabilities': array([0.5, 0.5, 0.5, 0.5]), 'brier_mean': 0.140625, 'unanimous_accuracy': 0.0, 'disagreement_probabilities': array([0.5, 0.5]), 'disagreement_brier': 0.03125}
heldout reward after training-pair fit: {'probabilities': array([0.8394109 , 0.93312322, 0.65921421, 0.06687678]), 'brier_mean': 0.022286160720660468, 'unanimous_accuracy': 1.0, 'disagreement_probabilities': array([0.65921421, 0.06687678]), 'disagreement_brier': 0.029441639707967603}
Synthetic, four-pair evaluation only; no human or language-quality claim.
PASS: posttraining-03
Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.
Reverse the comparison
# Experiment — Reverse the comparison: This checks label direction.
forward = float(preference_loss(w, chosen, rejected))
reverse = float(preference_loss(w, rejected, chosen))
assert reverse > forward
print('Forward/reversed preference NLL:', forward, reverse)Expected: The reversed-label loss is larger.
This checks label direction. A decreasing incorrectly signed loss can train a consistent but reversed preference model.
Fit conflicting labels conceptually
# Experiment — Fit conflicting labels conceptually
conflict = lambda margin: 0.5 * (
jax.nn.softplus(-margin) + jax.nn.softplus(margin)
)
np.testing.assert_allclose(conflict(0.0), np.log(2), atol=1e-6)
np.testing.assert_allclose(jax.grad(conflict)(0.0), 0.0, atol=1e-7)
assert float(conflict(2.0)) > float(conflict(0.0))
np.testing.assert_allclose(conflict(2.0), conflict(-2.0))
print('Balanced conflicting labels prefer a zero margin.')Expected: Zero margin minimizes the balanced contradictory-pair objective.
Disagreement is not fixed by flipping labels until training accuracy is high. The likelihood must represent the information actually present.
Make it yours
Where to put your code: Modify your own main.py, or use the Make it yours cell in your downloaded notebook. Run your attempt in the declared environment and verify your prediction before opening the reference solution.
Calculate the initial loss for tied reward scores independently and verify the trained ordering.
Show a writing guide and starter scaffold
How to write this exercise code
Learner-only inputs, outputs, checks and hints
Inputs
- Chosen and rejected feature pairs plus a zero-initialized reward vector.
- A trained reward vector for the same comparisons.
Outputs
- Initial tied-score preference loss and a check of the trained chosen-over-rejected ordering.
Invariants to check
- At equal scores, each binary preference contributes the same finite loss.
- The ordering check compares chosen and rejected scores for each labeled pair.
Hints
- Compute the score difference for each pair before applying the softplus loss.
- Keep the initial calculation independent from the trained ordering check.
import jax.numpy as jnp
def audit_initial_and_trained_preferences(chosen, rejected, trained_weights):
# TODO: calculate the tied-score loss and audit the trained pair ordering.
raise NotImplementedErrorReveal the reference solution
# Exercise solution
np.testing.assert_allclose(
preference_loss(jnp.zeros(2), chosen, rejected), -np.log(0.5), atol=1e-6
)
assert np.all(np.asarray((chosen - rejected) @ w) > 0)
print('Tied baseline and preference directions verified.')Show that feature scale can change scores without changing ranking
Transfer
Multiply the trained reward weights by two and compare ordering and preference probabilities.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- Feature pairs, the trained reward weights, and the same weights multiplied by two.
Outputs
- Score-order comparisons and preference probabilities before and after scaling.
Invariants to check
- Positive scaling preserves ordering while it changes probability sharpness.
Hints
- Compare score order and sigmoid probability as separate quantities.
import jax.numpy as jnp
def compare_reward_scale(features, chosen, rejected, weights):
# TODO: compare pair ordering and probabilities at both scales.
raise NotImplementedErrorShow a hint
Positive scaling preserves ordering but sharpens probabilities.
Reveal reference solution and reasoning
# Show that feature scale can change scores without changing ranking (Transfer): A policy’s reward-versus-KL tradeoff depends on reward scale.
scaled_rewards = features @ (2 * w)
np.testing.assert_array_equal(np.argsort(rewards), np.argsort(scaled_rewards))
assert float(preference_loss(2 * w, chosen, rejected)) < float(
preference_loss(w, chosen, rejected)
)
print('Ranking unchanged; confidence and downstream reward scale change.')A policy’s reward-versus-KL tradeoff depends on reward scale. Rank agreement alone is distinct from an equivalent RL objective.
Derive the linear reward gradient
Challenge
At a fresh weight vector, calculate the mean feature difference weighted by probability-minus-one and compare with autodiff. Repeat after reversing labels.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- Chosen and rejected feature matrices plus a fresh reward weight vector.
Outputs
- Mean feature-difference gradient from the likelihood formula and autodiff, repeated with reversed labels.
Invariants to check
- The gradient is averaged across comparisons and flips consistently when every label is reversed.
Hints
- Differentiate the scalar loss with respect to the weight vector.
- Use the same parameter values for both labelings.
import jax
import jax.numpy as jnp
def compare_linear_reward_gradient(weights, chosen, rejected):
# TODO: derive and check the preference-loss gradient before and after label reversal.
raise NotImplementedErrorShow a hint
Use NumPy sigmoid values for the independent calculation; reversing both feature order and probabilities is necessary.
Reveal reference solution and reasoning
# Derive the linear reward gradient (Challenge)
probe_w = jnp.array([0.2, -0.4])
for positive, negative in [(chosen, rejected), (rejected, chosen)]:
# Transfer the device result to the host for an independent NumPy comparison.
difference = np.asarray(positive - negative, dtype=np.float64)
margin_host = difference @ np.asarray(probe_w)
# Reduce along axis=0 to compute `analytic`.
analytic = np.mean(
(1 / (1 + np.exp(-margin_host)) - 1)[:, None] * difference, axis=0
)
np.testing.assert_allclose(
jax.grad(preference_loss)(probe_w, positive, negative),
analytic,
atol=1e-6,
)
print('Independent reward gradients agree in both label directions.')Checking both directions catches a sign error that a single decreasing training curve might conceal.
Evaluate held-out reward pairs and keep disagreement visible
Transfer
Define held-out feature pairs and their vote fractions before fitting. Calculate zero-weight baseline probabilities first. Fit the existing three training comparisons, then report held-out Brier score, unanimous-case accuracy, and the disagreement slice before and after. Keep the same pairs and labels for both measurements.
Show a writing guide and starter scaffold
How to write this practice solution
Learner-only inputs, outputs, checks and hints
Inputs
- The existing three training feature comparisons and fitted reward weights
- Four fixed held-out chosen/rejected feature pairs and vote fractions
- A declared slice label for unanimous and disagreement examples
Outputs
- Pre-fit baseline probabilities and Brier score
- Post-fit held-out Brier score and unanimous-case accuracy
- Separate disagreement probabilities and Brier score
Invariants to check
- Hold out the examples before fitting
- Use identical held-out pairs and vote fractions for baseline and post-fit metrics
- Score disagreement against its fractional vote; do not force a hard label
- Describe every score as a synthetic finite-fixture result
Hints
- Use the chosen-minus-rejected feature vector in the sigmoid margin.
- Compute baseline with
w_baseline = zeros_like(w)before the optimizer update. - Report unanimity and disagreement slices separately.
# Complete the baseline-first held-out report for fixed pair features.
raise NotImplementedErrorShow a hint
The probability that the chosen item wins is sigmoid((chosen - rejected) dot w). For a fractional vote v, Brier error is (p - v)^2. A disagreement pair does not have a single hard correctness label.
Reveal reference solution and reasoning
# Fixed synthetic held-out reward pairs. Baseline is calculated at w=0 before fitting.
heldout_chosen = jnp.array([[1.0, 0.0], [0.0, 1.0], [0.2, 0.2], [-1.0, -1.0]])
heldout_rejected = jnp.array([[0.0, 1.0], [-1.0, -1.0], [-0.1, 0.0], [0.0, 1.0]])
heldout_votes = np.array([1.0, 1.0, 0.5, 0.25], dtype=np.float64)
heldout_slices = np.array(["unanimous", "unanimous", "disagreement", "disagreement"])
# Define the baseline before any reward-model optimization.
w_baseline = jnp.zeros_like(w)
def heldout_report(weights):
margins = np.asarray((heldout_chosen - heldout_rejected) @ weights, dtype=np.float64)
probabilities = 1.0 / (1.0 + np.exp(-margins))
brier = (probabilities - heldout_votes) ** 2
unanimous = heldout_slices == "unanimous"
disagreement = ~unanimous
return {
"probabilities": probabilities,
"brier_mean": float(np.mean(brier)),
"unanimous_accuracy": float(np.mean((probabilities[unanimous] > 0.5) == (heldout_votes[unanimous] > 0.5))),
"disagreement_probabilities": probabilities[disagreement],
"disagreement_brier": float(np.mean(brier[disagreement])),
}
baseline = pretraining_heldout_baseline
after = heldout_report(w)
assert len(baseline["disagreement_probabilities"]) == 2
assert np.all(np.isfinite(after["probabilities"]))
print("heldout reward baseline:", baseline)
print("heldout reward after training-pair fit:", after)
print("Synthetic, four-pair evaluation only; no human or language-quality claim.")The baseline is explicitly tied to zero reward weights before fitting. The same held-out examples are scored after training. Unanimous accuracy and disagreement Brier are reported separately; the fractional vote remains a target probability, not a hard label. This is a finite synthetic mechanism check.
Diagnose the result
If margins have the wrong sign, inspect chosen/rejected order. If training ranking is perfect but held-out agreement fails, inspect prompt leakage, rubric consistency and reward-model overfitting.
Carry forward
- Pairwise preference loss depends only on score margins , starting at for tied scores and weighting feature differences by .
- Common additive reward offsets cancel in pairwise comparisons while positive weight rescaling sharpens confidence and alters the downstream reward-versus-KL balance.
Keep your evidence
Keep stable likelihood and gradient checks, contradictory preference controls, pre-fit and post-fit held-out probability/Brier metrics, separate disagreement slices and score-offset/scale limitations.
Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.
Saved in this browser. Export or back up your portfolio anytime in My learning.
Reference execution: CPU validation only. This worked reference is validated on CPU. Accelerator compatibility requires a separate source review and target execution receipt. Use the execution matrix to review device selection and data ownership before adapting a separate target program. Execution matrix and target qualification.