JAX PathwaysCheck your starting point
An episode stops because its time budget expires. Must its bootstrap value be zero?
Compare your reasoning
Not necessarily. Distinguish truncation from genuine terminal states and state whether the task is finite-horizon or continuing. A timeout alone does not imply that future value is zero.
If this is unfamiliar, work through Write a functional environment before attempting the project.
Build in stages
1. Test environment and rollout semantics
Check transitions independently, preserve terminal rewards and label time/environment axes. Store the behavior policy that collected each action.
2. Compare updates and evaluations fairly
Audit the clipped objective for both advantage signs. Freeze the evaluation protocol, compare a simple baseline and distinguish training seeds from repeated episodes.
Try a changed condition
Make a prediction before opening the reasoning. Keep your answer and the evidence you would collect.
You evaluate one trained agent over many episodes and report a tiny uncertainty interval. Does this measure sensitivity to training randomness?
Compare an approach
No. Those episodes characterize that agent under the evaluation distribution. Train independent agents to assess training variation, retain agent-level means and state the experimental unit.
Diagnose a likely failure
Symptom: PPO ratios always equal one after every update.
Check next: Check that old log probabilities were stored at collection rather than recomputed from the current policy.
Decide what is ready
Use policy-evaluation. Keep independent transition checks, behavior-policy identity, changed-horizon cases and paired evaluations across declared independent training seeds.
Where to put your project code
Write your code in: projects/policy-evaluation/my_model.py (copy projects/policy-evaluation/starter/model.py first: cp projects/policy-evaluation/starter/model.py projects/policy-evaluation/my_model.py).
Verify your implementation: python3 projects/policy-evaluation/tests/check.py --stage all --impl projects/policy-evaluation/my_model.py
Run the staged project checks on your implementation in projects/policy-evaluation/my_model.py and record your transfer experiment results in your portfolio.
What still needs further work
Continuous actions, value-function estimation and larger benchmark environments need new implementation and evaluation labs; this tabular task is deliberately bounded.