{ "schemaVersion": 1, "projects": [ { "schemaVersion": 1, "id": "regression-audit", "title": "Build a regression training audit", "pathwayId": "foundations", "status": "authored", "hardware": "CPU", "source": "projects/regression-audit/README.md", "requiredLessonIds": [ "welcome-01", "welcome-02", "arrays-01", "arrays-02", "arrays-03", "arrays-04", "first-gradient", "transforms-02", "transforms-03", "transforms-04", "transforms-05", "state-01", "state-02", "state-03", "state-04", "optimization-01", "optimization-02", "optimization-03", "optimization-04" ], "stages": [ { "id": "1", "title": "Prediction and objective", "command": "python3 projects/regression-audit/tests/check.py --stage 1", "evidence": "Shape contracts and independently checked scalar loss", "expected": "PASS stage 1: prediction, objective, and shape contract", "diagnosis": "If shapes differ, inspect the model output and target axes before changing the loss reduction.", "codeGuide": "In `projects/regression-audit/starter/model.py`, implement `predict(...)`, `loss(...)` using `contract`, `jnp.mean`, `predict`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def predict(params, x):\n # Step 1: Return `params['weight'] * x + params['bias']` to the caller.\n raise NotImplementedError('Stage 1: implement a linear prediction')\n\ndef loss(params, x, y):\n # Key APIs to use: `contract`, `jnp.mean`, `predict`\n # Step 1: Guard input contract (`x.shape != y.shape`) and fail fast if violated.\n # Step 2: Return `jnp.mean((predict(params, x) - y) ** 2)` to the caller.\n raise NotImplementedError('Stage 1: implement scalar MSE with shape checks')", "solutionSnippet": "# Function `predict` reference implementation:\ndef predict(params, x):\n # Return `params['weight'] * x + params['bias']` to the caller.\n return params['weight'] * x + params['bias']\n\n# Function `loss` reference implementation:\ndef loss(params, x, y):\n # Guard input contract (`x.shape != y.shape`) and fail fast if violated.\n if x.shape != y.shape:\n raise ValueError('Inputs and scalar targets must have matching shapes')\n # Return `jnp.mean((predict(params, x) - y) ** 2)` to the caller.\n return jnp.mean((predict(params, x) - y) ** 2)" }, { "id": "2", "title": "Gradient evidence", "command": "python3 projects/regression-audit/tests/check.py --stage 2", "evidence": "Autodiff and finite differences at two settings", "expected": "PASS stage 2: independent gradients at two parameter settings", "diagnosis": "If the derivative check fails, compare the stable independent loss and finite-difference tolerance. Preserve the same initialized parameters for both checks.", "codeGuide": "In `projects/regression-audit/starter/model.py`, implement `gradient_check(...)` using `jax.grad`, `loss`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def gradient_check(params, x, y, h=1e-2):\n # Key APIs to use: `jax.grad`, `loss`\n # Step 1: Differentiate the objective to obtain gradients `automatic`.\n # Step 2: Evaluate `finite` from the current inputs and state.\n # Step 3: Loop over `key` in `params`:\n # Step 4: Inside block: Evaluate `plus` from the current inputs and state.\n # Step 5: Inside block: Evaluate `minus` from the current inputs and state.\n # Step 6: Return `(automatic, finite)` to the caller.\n raise NotImplementedError('Stage 2: compare autodiff with independent differences')", "solutionSnippet": "# Function `gradient_check` reference implementation:\ndef gradient_check(params, x, y, h=1e-2):\n # Differentiate the objective to obtain `automatic` via automatic differentiation.\n automatic = jax.grad(loss)(params, x, y)\n # Construct dictionary `finite` with the structured fields for this stage.\n finite = {}\n # Iterate over `key` to step through the computation:\n for key in params:\n # Construct dictionary `plus` with the structured fields for this stage.\n plus = {**params, key: params[key] + h}\n # Construct dictionary `minus` with the structured fields for this stage.\n minus = {**params, key: params[key] - h}\n # Evaluate the compound expression for `finite[key]`.\n finite[key] = (loss(plus, x, y) - loss(minus, x, y)) / (2*h)\n # Return `(automatic, finite)` to the caller.\n return automatic, finite" }, { "id": "3", "title": "Compiled training", "command": "python3 projects/regression-audit/tests/check.py --stage 3", "evidence": "Recovered parameters and complete loss history", "expected": "PASS stage 3: compiled training and parameter recovery", "diagnosis": "If training fails, inspect the complete loss history and explicit state transition. Keep the failure output and compare held-out evaluation without mutating model state.", "codeGuide": "In `projects/regression-audit/starter/model.py`, implement `train(...)` using `contract`, `step`, `jax.value_and_grad`, `PyTree`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def train(params, x, y, rate=0.15, steps=200):\n # Key APIs to use: `contract`, `step`, `jax.value_and_grad`, `PyTree`, `tree.map`\n # Step 1: Guard input contract (`steps < 1`) and fail fast if violated.\n # Step 2: Function `step(p, _)` implementing this stage's computation:\n # Step 3: Return `jax.lax.scan(step, params, None, length=steps)` to the caller.\n raise NotImplementedError('Stage 3: carry parameters through a compiled loop')", "solutionSnippet": "# Function `train` reference implementation:\ndef train(params, x, y, rate=0.15, steps=200):\n # Guard input contract (`steps < 1`) and fail fast if violated.\n if steps < 1:\n raise ValueError('steps must be positive')\n # Define `step(p, _)` to evaluate the objective and its automatic derivatives:\n def step(p, _):\n # Differentiate the objective to obtain `(value, gradients)` via automatic differentiation.\n value, gradients = jax.value_and_grad(loss)(p, x, y)\n # Transform every leaf of the parameter PyTree (`updated`).\n updated = jax.tree.map(lambda v, g: v - rate * g, p, gradients)\n # Return `(updated, value)` to the caller.\n return updated, value\n # Return `jax.lax.scan(step, params, None, length=steps)` to the caller.\n return jax.lax.scan(step, params, None, length=steps)" }, { "id": "4", "title": "Training audit", "command": "python3 projects/regression-audit/tests/check.py --stage 4", "evidence": "Held-out performance, replay, and an unstable run", "expected": "PASS stage 4: held-out checks, replay, and failure diagnosis", "diagnosis": "If the audit fails, separate replay, held-out checks and deliberately unstable training; a low training loss alone cannot satisfy all three.", "codeGuide": "In `projects/regression-audit/starter/model.py`, implement `report(...)` using `train`, `jnp.array`, `fitted.items`, `loss`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 4 verifier.", "starterSnippet": "def report(params, x, y, rate=0.15, steps=200):\n # Key APIs to use: `train`, `jnp.array`, `fitted.items`, `loss`, `jnp.all`\n # Step 1: Run `train` to compute `(fitted, history)`.\n # Step 2: Create device-backed JAX array `heldout_x`.\n # Step 3: Evaluate `heldout_y` from the current inputs and state.\n # Step 4: Return `{'parameters': {k: float(v) for k, v in fitted.items()}, 'initial_loss': float(history[0]), 'training_loss': float(loss(fitted, x, y)), 'heldout_loss': float(loss(fitted, heldout_x, heldout_y)), 'finite': bool(jnp.all(jnp.isfinite(history))), 'steps': steps, 'learning_rate': rate, 'backend': jax.default_backend()}` to the caller.\n raise NotImplementedError('Stage 4: report fit, held-out error, and numerical health')", "solutionSnippet": "# Function `report` reference implementation:\ndef report(params, x, y, rate=0.15, steps=200):\n # Run `train` to compute `(fitted, history)`.\n fitted, history = train(params, x, y, rate, steps)\n # Initialize array `heldout_x` with explicit values and shape.\n heldout_x = jnp.array([-0.8, 0.2, 0.8])\n # Compute `heldout_y` as `2 * heldout_x + 1`.\n heldout_y = 2 * heldout_x + 1\n # Return `{'parameters': {k: float(v) for k, v in fitted.items()}, 'initial_loss': float(history[0]), 'training_loss': float(loss(fitted, x, y)), 'heldout_loss': float(loss(fitted, heldout_x, heldout_y)), 'finite': bool(jnp.all(jnp.isfinite(history))), 'steps': steps, 'learning_rate': rate, 'backend': jax.default_backend()}` to the caller.\n return {'parameters': {k: float(v) for k,v in fitted.items()},\n 'initial_loss': float(history[0]), 'training_loss': float(loss(fitted,x,y)),\n 'heldout_loss': float(loss(fitted,heldout_x,heldout_y)),\n 'finite': bool(jnp.all(jnp.isfinite(history))), 'steps': steps,\n 'learning_rate': rate, 'backend': jax.default_backend()}" } ], "rubric": [ "Implement matching prediction/target shapes and scalar loss", "Explain finite-difference tolerance and precision", "Recover known parameters with compiled state transitions", "Distinguish training fit from held-out checks", "Diagnose an unstable learning rate with recorded evidence" ], "setupNote": "Extract the ZIP into a fresh folder. That folder contains requirements-cpu.txt and projects/regression-audit. Use the extracted top folder as your working directory; a full course checkout is optional for this project. Open `projects/regression-audit/starter/model.py` in your editor, implement the stage functions directly inside that file, and verify each stage with `python3 projects/regression-audit/tests/check.py --stage 1` (through `--stage 4`).", "workspaceFile": "projects/regression-audit/starter/model.py", "referenceCommand": "python3 projects/regression-audit/tests/check.py --implementation solution --stage all", "assessmentId": "foundations" }, { "schemaVersion": 1, "id": "mlp-classifier", "title": "Build a reproducible classifier audit", "pathwayId": "models", "status": "authored", "hardware": "CPU", "source": "projects/mlp-classifier/README.md", "requiredLessonIds": [ "networks-01", "networks-02", "networks-03", "networks-04", "networks-05" ], "stages": [ { "id": "1", "title": "Model and shape contract", "command": "python3 projects/mlp-classifier/tests/check.py --stage 1 --implementation projects/mlp-classifier/my_model.py", "evidence": "Shape diagram, independent forward calculation, parameter count", "expected": "PASS stage 1: shape, wiring, parameter count, seeded initialization", "diagnosis": "If shapes differ, inspect the model output and target axes before changing the loss reduction.", "codeGuide": "In `projects/mlp-classifier/my_model.py`, implement `make_model(...)` using `Classifier`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def make_model(seed=0, width=8):\n \"\"\"Return an NNX module: Linear(2,width), tanh, Linear(width,1), (B,) logits.\"\"\"\n # Key APIs to use: `Classifier`\n # Step 1: Return `Classifier(seed, width)` to the caller.\n raise NotImplementedError('Stage 1: construct and verify the model')", "solutionSnippet": "# Function `make_model` reference implementation:\ndef make_model(seed=0, width=8):\n # Return `Classifier(seed, width)` to the caller.\n return Classifier(seed, width)" }, { "id": "2", "title": "Objective and derivative evidence", "command": "python3 projects/mlp-classifier/tests/check.py --stage 2 --implementation projects/mlp-classifier/my_model.py", "evidence": "Independent stable loss, two finite-difference checks, malformed-label diagnosis", "expected": "PASS stage 2: independent loss, two-seed finite differences, label contract", "diagnosis": "If the derivative check fails, compare the stable independent loss and finite-difference tolerance. Preserve the same initialized parameters for both checks.", "codeGuide": "In `projects/mlp-classifier/my_model.py`, implement `objective(...)` using `model`, `contract`, `shape`, `jnp.mean`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def objective(model, x, labels):\n \"\"\"Mean stable binary CE; reject logits/label shape mismatch.\"\"\"\n # Key APIs to use: `model`, `contract`, `shape`, `jnp.mean`, `optax.sigmoid_binary_cross_entropy`\n # Step 1: Run `model` to compute `scores`.\n # Step 2: Guard input contract (`scores.shape != labels.shape`) and fail fast if violated.\n # Step 3: Return `jnp.mean(optax.sigmoid_binary_cross_entropy(scores, labels))` to the caller.\n raise NotImplementedError('Stage 2: scalar loss and independent gradient evidence')", "solutionSnippet": "# Function `objective` reference implementation:\ndef objective(model, x, labels):\n # Run `model` to compute `scores`.\n scores = model(x)\n # Guard input contract (`scores.shape != labels.shape`) and fail fast if violated.\n if scores.shape != labels.shape:\n raise ValueError('scores and labels must both have shape (batch,)')\n # Return `jnp.mean(optax.sigmoid_binary_cross_entropy(scores, labels))` to the caller.\n return jnp.mean(optax.sigmoid_binary_cross_entropy(scores, labels))" }, { "id": "3", "title": "Training audit", "command": "python3 projects/mlp-classifier/tests/check.py --stage 3 --implementation projects/mlp-classifier/my_model.py", "evidence": "Held-out metrics, full replay, evaluation isolation, count-weighted aggregation", "expected": "PASS stage 3: held-out metrics, unequal-batch aggregation, evaluation isolation, full replay, changed seed", "diagnosis": "If training fails, inspect the complete loss history and explicit state transition. Keep the failure output and compare held-out evaluation without mutating model state.", "codeGuide": "In `projects/mlp-classifier/my_model.py`, implement `train(...)`, `evaluate(...)` using `contract`, `make_model`, `state`, `nnx.Optimizer`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def train(seed, x, labels, rate=.03, steps=200, width=8):\n \"\"\"Return (trained_model, pre_update_losses); compile updates, not evaluation.\"\"\"\n # Key APIs to use: `contract`, `make_model`, `state`, `nnx.Optimizer`, `optax.adam`\n # Step 1: Guard input contract (`steps < 1 or rate <= 0`) and fail fast if violated.\n # Step 2: Run `make_model` to compute `model`.\n # Step 3: Initialize the optimizer transformation and state (`optimizer`).\n # Step 4: Evaluate `losses` from the current inputs and state.\n # Step 5: Repeat the update loop over `range(steps)` steps:\n # Step 6: Inside block: Evaluate `_step(model, optimizer, x, labels)` and convert the result into Python scalar/collection `value`.\n raise NotImplementedError('Stage 3: reproducible training with Optax Adam')\n\ndef evaluate(model, x, labels):\n \"\"\"Return loss, accuracy, count without modifying any model state.\"\"\"\n # Key APIs to use: `model`, `objective`, `jnp.mean`\n # Step 1: Run `model` to compute `scores`.\n # Step 2: Evaluate `objective(model, x, labels)` and convert the result into Python scalar/collection `mean_loss`.\n # Step 3: Reduce across the target axis to summarize `accuracy`.\n # Step 4: Return `{'loss': mean_loss, 'accuracy': accuracy, 'count': len(labels)}` to the caller.\n raise NotImplementedError('Stage 3: isolated held-out metrics')", "solutionSnippet": "# Function `train` reference implementation:\ndef train(seed, x, labels, rate=.03, steps=200, width=8):\n # Guard input contract (`steps < 1 or rate <= 0`) and fail fast if violated.\n if steps < 1 or rate <= 0:\n raise ValueError('steps and rate must be positive')\n # Run `make_model` to compute `model`.\n model = make_model(seed, width)\n # Configure or step the Optax optimizer state (`optimizer`).\n optimizer = nnx.Optimizer(model, optax.adam(rate), wrt=nnx.Param)\n # Initialize list `losses` for the stage values.\n losses = []\n # Repeat the update loop over `range(steps)` steps:\n for _ in range(steps):\n # Evaluate `_step(model, optimizer, x, labels)` and convert the result into Python scalar/collection `value`.\n value = float(_step(model, optimizer, x, labels))\n # Guard input contract (`not np.isfinite(value)`) and fail fast if violated.\n if not np.isfinite(value):\n raise FloatingPointError('nonfinite pre-update training loss')\n # Append the current step result to `losses`.\n losses.append(value)\n # Return `(model, np.asarray(losses))` to the caller.\n return model, np.asarray(losses)\n\n# Function `evaluate` reference implementation:\ndef evaluate(model, x, labels):\n # Run `model` to compute `scores`.\n scores = model(x)\n # Evaluate `objective(model, x, labels)` and convert the result into Python scalar/collection `mean_loss`.\n mean_loss = float(objective(model, x, labels))\n # Aggregate array values to compute `accuracy`.\n accuracy = float(jnp.mean((scores > 0) == labels))\n # Return `{'loss': mean_loss, 'accuracy': accuracy, 'count': len(labels)}` to the caller.\n return {'loss': mean_loss, 'accuracy': accuracy, 'count': len(labels)}" } ], "rubric": [ "Explain the nonlinearity and parameter shapes", "Verify a stable scalar objective independently", "Compare a derivative at two initializations", "Keep training and held-out arrays separate", "Prove evaluation leaves model state unchanged", "Reproduce the complete update sequence", "Diagnose a deliberate failure and report the repair", "State the limitations of synthetic held-out evidence" ], "workspaceFile": "projects/mlp-classifier/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/mlp-classifier/starter/model.py projects/mlp-classifier/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/mlp-classifier/starter/model.py projects/mlp-classifier/my_model.py" } ], "referenceCommand": "python3 projects/mlp-classifier/tests/check.py --stage 3 --implementation solution", "setupNote": "Extract the ZIP into a fresh folder. That folder contains requirements-cpu.txt and projects/mlp-classifier. Use the extracted top folder as your working directory; a full course checkout is optional for this project. Copy the starter scaffold `projects/mlp-classifier/starter/model.py` to `projects/mlp-classifier/my_model.py`, implement the stage functions in `projects/mlp-classifier/my_model.py`, and verify each stage with `python3 projects/mlp-classifier/tests/check.py --implementation projects/mlp-classifier/my_model.py --stage 1` (through `--stage 3`).", "assessmentId": "models" }, { "schemaVersion": 1, "id": "scientific-inverse", "title": "Audit a differentiable scientific inverse problem", "pathwayId": "science", "status": "authored", "hardware": "CPU float64; no accelerator qualification", "source": "projects/scientific-inverse/README.md", "requiredLessonIds": [ "science-01", "science-02", "science-03", "science-04" ], "stages": [ { "id": "1", "title": "Simulation and numerical references", "command": "python3 projects/scientific-inverse/tests/check.py --implementation projects/scientific-inverse/my_model.py --stage 1", "evidence": "Two rate/grid comparisons, shape diagram and an unstable-step diagnosis", "expected": "PASS stage 1: two rates, grids, independent polynomial and exponential oracles, axes, invalid inputs", "diagnosis": "If the polynomial matches but the exponential does not, test grid refinement before changing the implementation.", "codeGuide": "In `projects/scientific-inverse/my_model.py`, implement `simulate(...)` using `contract`, `jnp.ndim`, `advance`, `lax.scan`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def simulate(rate, initials, steps=40, dt=0.05):\n \"\"\"RK4 cooling ensemble, shape (B, steps+1), including initial state.\n\n Require a nonempty vector of initials, scalar rate, positive integer steps\n and positive dt. Preserve JAX differentiation with respect to rate.\n \"\"\"\n # Key APIs to use: `contract`, `jnp.ndim`, `advance`, `lax.scan`, `jnp.concatenate`\n # Step 1: Guard input contract (`initials.ndim != 1 or initials.size == 0`) and fail fast if violated.\n # Step 2: Guard input contract (`not isinstance(steps, int) or steps < 1 or dt <= 0`) and fail fast if violated.\n # Step 3: Guard input contract (`jnp.ndim(rate) != 0`) and fail fast if violated.\n # Step 4: Function `advance(values, unused)` implementing this stage's computation:\n # Step 5: Run a compiled sequential scan over the time/step axis (`(_, tail)`).\n # Step 6: Return `jnp.concatenate((initials[None, :], tail), axis=0).T` to the caller.\n raise NotImplementedError('Stage 1: explicit RK4 transition and scan')", "solutionSnippet": "# Function `simulate` reference implementation:\ndef simulate(rate, initials, steps=40, dt=0.05):\n \"\"\"Return batch-major states, including the initial observation.\"\"\"\n # Guard input contract (`initials.ndim != 1 or initials.size == 0`) and fail fast if violated.\n if initials.ndim != 1 or initials.size == 0:\n raise ValueError('initials must be a nonempty vector')\n # Guard input contract (`not isinstance(steps, int) or steps < 1 or dt <= 0`) and fail fast if violated.\n if not isinstance(steps, int) or steps < 1 or dt <= 0:\n raise ValueError('steps must be a positive integer and dt must be positive')\n # Guard input contract (`jnp.ndim(rate) != 0`) and fail fast if violated.\n if jnp.ndim(rate) != 0:\n raise ValueError('rate must be scalar')\n # Function `advance(values, unused)` implementing this stage's computation:\n def advance(values, unused):\n # Compute `a` as `-rate*values`.\n a = -rate*values\n # Compute `b` as `-rate*(values+dt*a/2)`.\n b = -rate*(values+dt*a/2)\n # Compute `c` as `-rate*(values+dt*b/2)`.\n c = -rate*(values+dt*b/2)\n # Compute `d` as `-rate*(values+dt*c)`.\n d = -rate*(values+dt*c)\n # Compute `next_values` as `values+dt*(a+2*b+2*c+d)/6`.\n next_values = values+dt*(a+2*b+2*c+d)/6\n # Return `(next_values, next_values)` to the caller.\n return next_values, next_values\n # Run compiled structured control flow via `jax.lax` (`(_, tail)`).\n _, tail = jax.lax.scan(advance, initials, None, length=steps)\n # Return `jnp.concatenate((initials[None, :], tail), axis=0).T` to the caller.\n return jnp.concatenate((initials[None,:],tail),axis=0).T" }, { "id": "2", "title": "Objective and solver sensitivity", "command": "python3 projects/scientific-inverse/tests/check.py --implementation projects/scientific-inverse/my_model.py --stage 2", "evidence": "Independent scalar loss, two finite-difference probes and malformed-observation checks", "expected": "PASS stage 2: independent objective, log-rate finite differences at two probes, malformed-target rejection", "diagnosis": "If the derivative is zero but finite differences are nonzero, inspect detached arrays or missing dependence on log-rate.", "codeGuide": "In `projects/scientific-inverse/my_model.py`, implement `loss(...)` using `contract`, `shape`, `simulate`, `jnp.exp`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def loss(log_rate, initials, observations, dt=0.05):\n \"\"\"Mean squared observation error using rate=exp(log_rate).\n\n Reject observations not shaped (B, steps+1) with at least two times.\n \"\"\"\n # Key APIs to use: `contract`, `shape`, `simulate`, `jnp.exp`, `jnp.mean`\n # Step 1: Guard input contract (`observations.ndim != 2 or observations.shape[0] != len(initials) or observations.shape[1] < 2`) and fail fast if violated.\n # Step 2: Run `simulate` to compute `predictions`.\n # Step 3: Return `jnp.mean((predictions - observations) ** 2)` to the caller.\n raise NotImplementedError('Stage 2: scalar differentiable observation loss')", "solutionSnippet": "# Function `loss` reference implementation:\ndef loss(log_rate, initials, observations, dt=0.05):\n # Guard input contract (`observations.ndim != 2 or observations.shape[0] != len(initials) or observations.shape[1] < 2`) and fail fast if violated.\n if observations.ndim != 2 or observations.shape[0] != len(initials) or observations.shape[1] < 2:\n raise ValueError('observations must have shape (batch, steps+1) with at least two times')\n # Run `simulate` to compute `predictions`.\n predictions = simulate(jnp.exp(log_rate),initials,int(observations.shape[1]-1),dt)\n # Return `jnp.mean((predictions - observations) ** 2)` to the caller.\n return jnp.mean((predictions-observations)**2)" }, { "id": "3", "title": "Parameter recovery and an inference report", "command": "python3 projects/scientific-inverse/tests/check.py --implementation projects/scientific-inverse/my_model.py --stage 3", "evidence": "Changed physical rates, frozen held-out metrics, replay, refinement and noisy search comparison", "expected": "PASS stage 3: changed-parameter recovery, frozen held-out metrics, replay, refinement, noisy independent grid-search agreement", "diagnosis": "If fit is excellent but inferred rate changes with the grid, quantify numerical bias; do not tune against held-out observations.", "codeGuide": "In `projects/scientific-inverse/my_model.py`, implement `fit(...)`, `evaluate(...)` using `contract`, `jnp.log`, `jnp.asarray`, `loss`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def fit(initials, observations, dt=0.05, initial_rate=1.4, updates=250, learning_rate=0.4):\n \"\"\"Return positive fitted rate and loss history including pre-update loss.\n\n Use exactly updates gradient descent updates in log-rate, return a history\n of length updates+1, and reject nonpositive fitting configuration values.\n \"\"\"\n # Key APIs to use: `contract`, `jnp.log`, `jnp.asarray`, `loss`, `objective`\n # Step 1: Guard input contract (`initial_rate <= 0 or updates < 1 or learning_rate <= 0`) and fail fast if violated.\n # Step 2: Create device-backed JAX array `theta`.\n # Step 3: Evaluate `objective` from the current inputs and state.\n # Step 4: Evaluate `history` from the current inputs and state.\n # Step 5: Differentiate the objective to obtain gradients `update`.\n # Step 6: Repeat the update loop over `range(updates)` steps:\n raise NotImplementedError('Stage 3: reproducible positive-parameter fit')\n\ndef evaluate(rate, initials, observations, dt=0.05):\n \"\"\"Return mse, rmse, max_abs_error, trajectory_count, observation_count.\n\n Use observations only for measurement\n do not fit or mutate inputs.\n Reject nonfinite/nonpositive rate.\n \"\"\"\n # Key APIs to use: `contract`, `np.isfinite`, `loss`, `jnp.log`, `simulate`\n # Step 1: Guard input contract (`not np.isfinite(rate) or rate <= 0`) and fail fast if violated.\n # Step 2: Run `loss` to compute `value`.\n # Step 3: Run `simulate` to compute `predictions`.\n # Step 4: Evaluate `error` from the current inputs and state.\n # Step 5: Return `{'mse': float(value), 'rmse': float(jnp.sqrt(value)), 'max_abs_error': float(jnp.max(jnp.abs(error))), 'trajectory_count': int(len(initials)), 'observation_count': int(observations.size)}` to the caller.\n raise NotImplementedError('Stage 3: isolated evaluation')", "solutionSnippet": "# Function `fit` reference implementation:\ndef fit(initials, observations, dt=0.05, initial_rate=1.4, updates=250, learning_rate=0.4):\n # Guard input contract (`initial_rate <= 0 or updates < 1 or learning_rate <= 0`) and fail fast if violated.\n if initial_rate <= 0 or updates < 1 or learning_rate <= 0:\n raise ValueError('initial rate, update count and learning rate must be positive')\n # Create device-backed JAX array `theta`.\n theta = jnp.log(jnp.asarray(initial_rate,dtype=jnp.float64))\n # Compute `objective` as `lambda p: loss(p,initials,observations,dt)`.\n objective = lambda p: loss(p,initials,observations,dt)\n # Validate before entering the compiled update and retain a pre-update point.\n history = [float(objective(theta))]\n # Differentiate the objective to obtain `update` via automatic differentiation.\n update = jax.jit(lambda p:p-learning_rate*jax.grad(objective)(p))\n # Repeat the update loop over `range(updates)` steps:\n for _ in range(updates):\n # Run `update` to compute `theta`.\n theta=update(theta)\n # Append the current step result to `history`.\n history.append(float(objective(theta)))\n # Return `(float(jnp.exp(theta)), np.asarray(history))` to the caller.\n return float(jnp.exp(theta)),np.asarray(history)\n\n# Function `evaluate` reference implementation:\ndef evaluate(rate, initials, observations, dt=0.05):\n # Guard input contract (`not np.isfinite(rate) or rate <= 0`) and fail fast if violated.\n if not np.isfinite(rate) or rate <= 0:\n raise ValueError('rate must be finite and positive')\n # Run `loss` to compute `value`.\n value=loss(jnp.log(rate),initials,observations,dt)\n # Run `simulate` to compute `predictions`.\n predictions=simulate(rate,initials,int(observations.shape[1]-1),dt)\n # Compute `error` as `predictions-observations`.\n error=predictions-observations\n # Return `{'mse': float(value), 'rmse': float(jnp.sqrt(value)), 'max_abs_error': float(jnp.max(jnp.abs(error))), 'trajectory_count': int(len(initials)), 'observation_count': int(observations.size)}` to the caller.\n return {'mse':float(value),'rmse':float(jnp.sqrt(value)),\n 'max_abs_error':float(jnp.max(jnp.abs(error))),\n 'trajectory_count':int(len(initials)),'observation_count':int(observations.size)}" } ], "rubric": [ "Separate discrete algorithm correctness from continuous approximation error", "Verify batch/time axes, singleton batches and malformed-input rejection", "Check rate sensitivities against independent finite differences at two probes", "Recover changed physical rates from independently generated observations", "Preserve held-out data and prove deterministic replay", "Compare noisy-data inference with an independent search", "Report grid refinement, non-identifiability and limits of synthetic evidence" ], "workspaceFile": "projects/scientific-inverse/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/scientific-inverse/starter/model.py projects/scientific-inverse/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/scientific-inverse/starter/model.py projects/scientific-inverse/my_model.py" } ], "referenceCommand": "python3 projects/scientific-inverse/tests/check.py --implementation solution --stage all", "setupNote": "Extract the project ZIP into a fresh folder, install its requirements-cpu.txt in your course environment, and run commands from the extracted top folder. CPU float64 is enabled by the scaffold. Copy the starter scaffold `projects/scientific-inverse/starter/model.py` to `projects/scientific-inverse/my_model.py`, implement the stage functions in `projects/scientific-inverse/my_model.py`, and verify each stage with `python3 projects/scientific-inverse/tests/check.py --implementation projects/scientific-inverse/my_model.py --stage 1` (through `--stage 3`).", "assessmentId": "science" }, { "schemaVersion": 1, "id": "policy-evaluation", "title": "Train and audit a policy from interaction", "pathwayId": "rl", "status": "authored", "hardware": "CPU", "source": "projects/policy-evaluation/README.md", "requiredLessonIds": [ "rl-01", "rl-02", "rl-03", "rl-04" ], "stages": [ { "id": "1", "title": "Functional environment", "command": "python3 projects/policy-evaluation/tests/check.py --stage 1 --implementation projects/policy-evaluation/my_model.py", "evidence": "Transition table and terminal/timeout trace", "expected": "PASS stage 1", "diagnosis": "Inspect boundary moves, deadline priority and rewards after done.", "codeGuide": "In `projects/policy-evaluation/my_model.py`, implement `reset(...)`, `step(...)` using `State`, `random.randint`, `jnp.int32`, `jnp.bool_`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def reset(key):\n \"\"\"Stage 1: uniform start at 0 or 1, zero elapsed, unfinished.\"\"\"\n # Key APIs to use: `State`, `random.randint`, `jnp.int32`, `jnp.bool_`\n # Step 1: Return `State(jax.random.randint(key, (), 0, 2), jnp.int32(0), jnp.bool_(False))` to the caller.\n raise NotImplementedError('Implement explicit reset state')\n\ndef step(state, action, horizon=8):\n \"\"\"Stage 1: return (State, reward, terminated event, truncated event).\n\n Binary actions move left/right\n boundaries are 0 and goal 3. Active moves\n pay -.01 except entering the goal pays 1. Done states absorb with no reward.\n \"\"\"\n # Key APIs to use: `jnp.clip`, `jnp.where`, `active.astype`, `State`\n # Step 1: Evaluate `active` from the current inputs and state.\n # Step 2: Combine or mask array elements to form `proposed`.\n # Step 3: Combine or mask array elements to form `position`.\n # Step 4: Evaluate `elapsed` from the current inputs and state.\n # Step 5: Evaluate `terminated` from the current inputs and state.\n # Step 6: Evaluate `truncated` from the current inputs and state.\n raise NotImplementedError('Implement and enumerate transition rules')", "solutionSnippet": "# Function `reset` reference implementation:\ndef reset(key):\n \"\"\"Start uniformly at position 0 or 1; goal is position 3.\"\"\"\n # Return `State(jax.random.randint(key, (), 0, 2), jnp.int32(0), jnp.bool_(False))` to the caller.\n return State(jax.random.randint(key, (), 0, 2), jnp.int32(0), jnp.bool_(False))\n\n# Function `step` reference implementation:\ndef step(state, action, horizon=8):\n \"\"\"Action 0: left, 1: right. Caller supplies binary action and positive horizon.\n\n Returns next state, reward, termination event, truncation event. Finished\n states absorb without more rewards/events. No implicit reset or lost final observation.\n \"\"\"\n # Compute `active` as `~state.done`.\n active = ~state.done\n # Combine or mask array elements to form `proposed`.\n proposed = jnp.clip(state.position + 2 * action - 1, 0, 3)\n # Combine or mask array elements to form `position`.\n position = jnp.where(active, proposed, state.position)\n # Compute `elapsed` as `state.elapsed + active.astype(jnp.int32)`.\n elapsed = state.elapsed + active.astype(jnp.int32)\n # Compute `terminated` as `active & (position == 3)`.\n terminated = active & (position == 3)\n # Compute `truncated` as `active & ~terminated & (elapsed >= horizon)`.\n truncated = active & ~terminated & (elapsed >= horizon)\n # Combine or mask array elements to form `reward`.\n reward = jnp.where(active, jnp.where(terminated, 1., -.01), 0.)\n # Return `(State(position, elapsed, state.done | terminated | truncated), reward, terminated, truncated)` to the caller.\n return State(position, elapsed, state.done | terminated | truncated), reward, terminated, truncated" }, { "id": "2", "title": "Batched trajectories", "command": "python3 projects/policy-evaluation/tests/check.py --stage 2 --implementation projects/policy-evaluation/my_model.py", "evidence": "Reconstructed episodes, axis diagram and key replay", "expected": "PASS stage 2", "diagnosis": "Check that active describes the state before the successful transition.", "codeGuide": "In `projects/policy-evaluation/my_model.py`, implement `log_probability(...)`, `rollout(...)`, `returns_to_go(...)` using `jnp.minimum`, `jnp.where`, `nn.log_sigmoid`, `key`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def log_probability(theta, observations, actions):\n \"\"\"Stage 2: theta contains one right-action logit for positions 0, 1, 2.\"\"\"\n # Key APIs to use: `jnp.minimum`, `jnp.where`, `nn.log_sigmoid`\n # Step 1: Reduce across the target axis to summarize `logits`.\n # Step 2: Return `jnp.where(actions == 1, jax.nn.log_sigmoid(logits), jax.nn.log_sigmoid(-logits))` to the caller.\n raise NotImplementedError('Use stable log-sigmoid probabilities')\n\ndef rollout(theta, key, batch_size=256, horizon=8):\n \"\"\"Stage 2: final State and time-major record dictionary.\n\n Fields: observation, action, reward, active, terminated, truncated, old_logp.\n Use fresh keys across reset, time, and environments. Do not auto-reset.\n \"\"\"\n # Key APIs to use: `key`, `random.split`, `jax.vmap`, `body`, `subkeys`\n # Step 1: Split the PRNG key deterministically into independent subkeys (`(reset_key, action_key)`).\n # Step 2: Split the PRNG key deterministically into independent subkeys (`states`).\n # Step 3: Split the PRNG key deterministically into independent subkeys (`time_keys`).\n # Step 4: Function `body(states, time_key)` implementing this stage's computation:\n # Step 5: Run a compiled sequential scan over the time/step axis (`(final, records)`).\n # Step 6: Return `(final, records)` to the caller.\n raise NotImplementedError('Build vmap within scan')\n\ndef returns_to_go(rewards):\n \"\"\"Stage 2: sum future rewards along time axis, preserving both axes.\"\"\"\n # Key APIs to use: `jnp.cumsum`\n # Step 1: Return `jnp.cumsum(rewards[::-1], axis=0)[::-1]` to the caller.\n raise NotImplementedError('Implement reverse cumulative returns')", "solutionSnippet": "# Function `log_probability` reference implementation:\ndef log_probability(theta, observations, actions):\n # Reduce across the target axis to summarize `logits`.\n logits = theta[jnp.minimum(observations, 2)]\n # Return `jnp.where(actions == 1, jax.nn.log_sigmoid(logits), jax.nn.log_sigmoid(-logits))` to the caller.\n return jnp.where(actions == 1, jax.nn.log_sigmoid(logits), jax.nn.log_sigmoid(-logits))\n\n# Function `rollout` reference implementation:\ndef rollout(theta, key, batch_size=256, horizon=8):\n # Create or split explicit PRNG key(s) (`(reset_key, action_key)`) for reproducible randomness.\n reset_key, action_key = jax.random.split(key)\n # Create or split explicit PRNG key(s) (`states`) for reproducible randomness.\n states = jax.vmap(reset)(jax.random.split(reset_key, batch_size))\n # Create or split explicit PRNG key(s) (`time_keys`) for reproducible randomness.\n time_keys = jax.random.split(action_key, horizon)\n # Function `body(states, time_key)` implementing this stage's computation:\n def body(states, time_key):\n # Compute `observations` as `states.position`.\n observations = states.position\n # Split the PRNG key deterministically into independent subkeys (`keys`).\n keys = jax.random.split(time_key, batch_size)\n # Reduce across the target axis to summarize `probabilities`.\n probabilities = jax.nn.sigmoid(theta[jnp.minimum(observations, 2)])\n # Draw pseudorandom samples for `actions` using the explicit RNG state.\n actions = jax.vmap(jax.random.bernoulli)(keys, probabilities).astype(jnp.int32)\n # Vectorize across the batch dimension without a Python loop (`(next_states, rewards, terminated, truncated)`).\n next_states, rewards, terminated, truncated = jax.vmap(\n lambda s, a: step(s, a, horizon))(states, actions)\n # Evaluate `observation=observations, action=actions, reward=rewards, active=~states.done, terminated=terminated, truncated=truncated, old_logp=log_probability(theta, observations, actions)` and convert the result into Python scalar/collection `record`.\n record = dict(observation=observations, action=actions, reward=rewards,\n active=~states.done, terminated=terminated, truncated=truncated,\n old_logp=log_probability(theta, observations, actions))\n # Return `(next_states, record)` to the caller.\n return next_states, record\n # Run a compiled sequential scan over the time/step axis (`(final, records)`).\n final, records = jax.lax.scan(body, states, time_keys)\n # Return `(final, records)` to the caller.\n return final, records\n\n# Function `returns_to_go` reference implementation:\ndef returns_to_go(rewards):\n \"\"\"Undiscounted finite-episode objective; rewards after done are already zero.\"\"\"\n # Return `jnp.cumsum(rewards[::-1], axis=0)[::-1]` to the caller.\n return jnp.cumsum(rewards[::-1], axis=0)[::-1]" }, { "id": "3", "title": "Policy objectives", "command": "python3 projects/policy-evaluation/tests/check.py --stage 3 --implementation projects/policy-evaluation/my_model.py", "evidence": "Independent enumeration, finite differences and sampled gradient comparison", "expected": "PASS stage 3", "diagnosis": "Keep behavior log probabilities frozen and check both signs of the clipped objective.", "codeGuide": "In `projects/policy-evaluation/my_model.py`, implement `policy_loss(...)`, `exact_return(...)` using `lax.stop_gradient`, `returns_to_go`, `log_probability`, `jnp.exp`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def policy_loss(theta, records, method='ppo', clip=.2):\n \"\"\"Stage 3: negative live-action sum divided by episode count.\n\n REINFORCE uses log probability times frozen reward-to-go. PPO uses the\n minimum of ratio*target and clipped-ratio*target, with frozen old logp.\n \"\"\"\n # Key APIs to use: `lax.stop_gradient`, `returns_to_go`, `log_probability`, `jnp.exp`, `jnp.minimum`\n # Step 1: Run `jax.lax.stop_gradient` to compute `advantage`.\n # Step 2: Run `jax.lax.stop_gradient` to compute `old_logp`.\n # Step 3: Run `log_probability` to compute `new_logp`.\n # Step 4: Branch on condition `method == 'reinforce'`:\n # Step 5: Return `-jnp.sum(jnp.where(records['active'], terms, 0.0)) / records['reward'].shape[1]` to the caller.\n raise NotImplementedError('Implement both objective branches')\n\ndef exact_return(theta, horizon=8):\n \"\"\"Stage 3: exact finite-horizon expectation, averaged over starts 0 and 1.\"\"\"\n # Key APIs to use: `nn.sigmoid`, `jnp.arange`, `jnp.maximum`, `backup`, `jnp.where`\n # Step 1: Run `jax.nn.sigmoid` to compute `p`.\n # Step 2: Create evenly spaced index values in `states`.\n # Step 3: Reduce across the target axis to summarize `left`.\n # Step 4: Evaluate `right` from the current inputs and state.\n # Step 5: Function `backup(_, values)` implementing this stage's computation:\n # Step 6: Allocate initialized array `values` with the specified shape and dtype.\n raise NotImplementedError('Use dynamic programming, not random samples')", "solutionSnippet": "# Function `policy_loss` reference implementation:\ndef policy_loss(theta, records, method='ppo', clip=.2):\n \"\"\"Normalize by episodes, not live transitions. Freeze behavior data/returns.\"\"\"\n # Run `jax.lax.stop_gradient` to compute `advantage`.\n advantage = jax.lax.stop_gradient(returns_to_go(records['reward']))\n # Run `jax.lax.stop_gradient` to compute `old_logp`.\n old_logp = jax.lax.stop_gradient(records['old_logp'])\n # Run `log_probability` to compute `new_logp`.\n new_logp = log_probability(theta, records['observation'], records['action'])\n # Branch on condition `method == 'reinforce'`:\n if method == 'reinforce':\n terms = new_logp * advantage\n elif method == 'ppo':\n ratio = jnp.exp(new_logp - old_logp)\n terms = jnp.minimum(ratio * advantage, jnp.clip(ratio, 1-clip, 1+clip) * advantage)\n else:\n raise ValueError('method must be reinforce or ppo')\n # Return `-jnp.sum(jnp.where(records['active'], terms, 0.0)) / records['reward'].shape[1]` to the caller.\n return -jnp.sum(jnp.where(records['active'], terms, 0.)) / records['reward'].shape[1]\n\n# Function `exact_return` reference implementation:\ndef exact_return(theta, horizon=8):\n \"\"\"Dynamic programming over states, independent of sampled rollouts.\"\"\"\n # Run `jax.nn.sigmoid` to compute `p`.\n p = jax.nn.sigmoid(theta)\n # Initialize array `states` with explicit values and shape.\n states = jnp.arange(3)\n # Reduce across the target axis to summarize `left`.\n left = jnp.maximum(states-1, 0)\n # Compute `right` as `states+1`.\n right = states+1\n # Function `backup(_, values)` implementing this stage's computation:\n def backup(_, values):\n # Compute `q_left` as `-.01 + values[left]`.\n q_left = -.01 + values[left]\n # Combine or mask array elements to form `q_right`.\n q_right = jnp.where(right == 3, 1., -.01 + values[right])\n # Return `jnp.concatenate(((1 - p) * q_left + p * q_right, jnp.zeros(1)))` to the caller.\n return jnp.concatenate(((1-p)*q_left+p*q_right, jnp.zeros(1)))\n # Allocate initialized array `values` with the specified shape and dtype.\n values = jax.lax.fori_loop(0, horizon, backup, jnp.zeros(4))\n # Return `jnp.mean(values[:2])` to the caller.\n return jnp.mean(values[:2])" }, { "id": "4", "title": "Training and held-out evaluation", "command": "python3 projects/policy-evaluation/tests/check.py --stage 4 --implementation projects/policy-evaluation/my_model.py", "evidence": "Multi-seed learning histories, baselines and per-agent held-out report", "expected": "PASS stage 4", "diagnosis": "Separate training keys from evaluation streams and preserve raw returns.", "codeGuide": "In `projects/policy-evaluation/my_model.py`, implement `train(...)`, `evaluate(...)` using `contract`, `or`, `in`, `_train`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 4 verifier.", "starterSnippet": "def train(seed=0, updates=40, batch_size=256, horizon=8, method='ppo', epochs=3, learning_rate=.5):\n \"\"\"Stage 4: (final theta, exact-return history including initial value).\n\n Start at zero logits. Collect fresh data for each update. Freeze each batch\n across its epochs. REINFORCE requires one epoch per fresh batch.\n \"\"\"\n # Key APIs to use: `contract`, `or`, `in`, `_train`\n # Step 1: Guard input contract (`updates < 1 or batch_size < 1 or horizon < 1 or (epochs < 1) or (learning_rate <= 0)`) and fail fast if violated.\n # Step 2: Guard input contract (`method not in ('ppo', 'reinforce')`) and fail fast if violated.\n # Step 3: Guard input contract (`method == 'reinforce' and epochs != 1`) and fail fast if violated.\n # Step 4: Return `_train(seed, updates, batch_size, horizon, method, epochs, learning_rate)` to the caller.\n raise NotImplementedError('Implement complete rollout and update cycle')\n\ndef evaluate(theta, seeds, batch_size=512, horizon=8):\n \"\"\"Stage 4: reject malformed theta, duplicate/fewer-than-two seeds or bad sizes.\n\n Return seeds, episode_returns (streams,batch), stream_means, mean and exact.\n Keep policy immutable and randomness separate from training.\n \"\"\"\n # Key APIs to use: `jnp.asarray`, `contract`, `jnp.all`, `jnp.isfinite`, `or`\n # Step 1: Create device-backed JAX array `theta`.\n # Step 2: Guard input contract (`theta.shape != (3,) or not bool(jnp.all(jnp.isfinite(theta)))`) and fail fast if violated.\n # Step 3: Evaluate `seeds` and convert the result into Python scalar/collection `seeds`.\n # Step 4: Guard input contract (`len(seeds) < 2 or len(set(seeds)) != len(seeds) or batch_size < 1 or (horizon < 1)`) and fail fast if violated.\n # Step 5: Convert `returns` to a host NumPy array for inspection or verification.\n # Step 6: Return `{'seeds': list(seeds), 'episode_returns': returns, 'stream_means': returns.mean(1), 'mean': float(returns.mean()), 'exact': float(exact_return(theta, horizon))}` to the caller.\n raise NotImplementedError('Retain disaggregated held-out evaluation')", "solutionSnippet": "# Function `train` reference implementation:\ndef train(seed=0, updates=40, batch_size=256, horizon=8, method='ppo', epochs=3, learning_rate=.5):\n # Guard input contract (`updates < 1 or batch_size < 1 or horizon < 1 or (epochs < 1) or (learning_rate <= 0)`) and fail fast if violated.\n if updates < 1 or batch_size < 1 or horizon < 1 or epochs < 1 or learning_rate <= 0:\n raise ValueError('positive updates, batch_size, horizon, epochs and learning_rate required')\n # Guard input contract (`method not in ('ppo', 'reinforce')`) and fail fast if violated.\n if method not in ('ppo', 'reinforce'):\n raise ValueError('unknown method')\n # Guard input contract (`method == 'reinforce' and epochs != 1`) and fail fast if violated.\n if method == 'reinforce' and epochs != 1:\n raise ValueError('REINFORCE uses one update per fresh on-policy rollout')\n # Return `_train(seed, updates, batch_size, horizon, method, epochs, learning_rate)` to the caller.\n return _train(seed, updates, batch_size, horizon, method, epochs, learning_rate)\n\n# Function `evaluate` reference implementation:\ndef evaluate(theta, seeds, batch_size=512, horizon=8):\n \"\"\"One mean per evaluation stream; preserve raw episode returns too.\"\"\"\n # Create device-backed JAX array `theta`.\n theta = jnp.asarray(theta)\n # Guard input contract (`theta.shape != (3,) or not bool(jnp.all(jnp.isfinite(theta)))`) and fail fast if violated.\n if theta.shape != (3,) or not bool(jnp.all(jnp.isfinite(theta))):\n raise ValueError('theta must contain three finite logits')\n # Evaluate `seeds` and convert the result into Python scalar/collection `seeds`.\n seeds = tuple(seeds)\n # Guard input contract (`len(seeds) < 2 or len(set(seeds)) != len(seeds) or batch_size < 1 or (horizon < 1)`) and fail fast if violated.\n if len(seeds) < 2 or len(set(seeds)) != len(seeds) or batch_size < 1 or horizon < 1:\n raise ValueError('provide at least two distinct evaluation seeds and positive sizes')\n # Convert `returns` to a host NumPy array for inspection or verification.\n returns = np.stack([np.asarray(rollout(theta, jax.random.key(s), batch_size, horizon)[1]['reward'].sum(0)) for s in seeds])\n # Return `{'seeds': list(seeds), 'episode_returns': returns, 'stream_means': returns.mean(1), 'mean': float(returns.mean()), 'exact': float(exact_return(theta, horizon))}` to the caller.\n return {'seeds': list(seeds), 'episode_returns': returns, 'stream_means': returns.mean(1),\n 'mean': float(returns.mean()), 'exact': float(exact_return(theta, horizon))}" } ], "rubric": [ "Enumerate transition and episode-ending contracts independently", "Keep reset and final observations explicit", "Collect actual randomized trajectories with reproducible independent keys", "Derive the policy gradient and check it against an exact expectation", "Keep old probabilities and targets frozen during clipped updates", "Train multiple agents from fresh interaction", "Evaluate with held-out streams and preserve per-agent variability", "State the fixture limitations and diagnose one failed invariant" ], "workspaceFile": "projects/policy-evaluation/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/policy-evaluation/starter/model.py projects/policy-evaluation/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/policy-evaluation/starter/model.py projects/policy-evaluation/my_model.py" } ], "referenceCommand": "python3 projects/policy-evaluation/tests/check.py --stage 4 --implementation solution", "setupNote": "Extract the project ZIP and work from its top folder, which contains requirements-cpu.txt and projects/policy-evaluation. A full checkout is optional. Copy the starter scaffold `projects/policy-evaluation/starter/model.py` to `projects/policy-evaluation/my_model.py`, implement the stage functions in `projects/policy-evaluation/my_model.py`, and verify each stage with `python3 projects/policy-evaluation/tests/check.py --implementation projects/policy-evaluation/my_model.py --stage 1` (through `--stage 4`).", "assessmentId": "rl" }, { "schemaVersion": 1, "id": "bayesian-regression", "title": "Audit a Bayesian regression model", "pathwayId": "probability", "status": "authored", "hardware": "CPU", "source": "projects/bayesian-regression/README.md", "requiredLessonIds": [ "probability-01", "probability-02", "probability-03", "probability-04" ], "stages": [ { "id": "1", "title": "Exact posterior and density", "command": "python3 projects/bayesian-regression/tests/check.py --stage 1 --implementation projects/bayesian-regression/my_model.py", "evidence": "Model statement, posterior derivation, independent NumPy oracle and gradient check", "expected": "PASS stage 1: posterior oracle, non-centered/rank-deficient data, log density, derivatives, input contracts", "diagnosis": "Check target shape and scale versus variance conventions before changing the solver.", "codeGuide": "In `projects/bayesian-regression/my_model.py`, implement `fit(...)`, `log_joint(...)` using `jnp.asarray`, `contract`, `or`, `be`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def fit(X, y, noise_scale=1., prior_scale=2.):\n \"\"\"Stage 1: validate arrays; return Gaussian posterior (mean, covariance).\n X: (observations, features), y: (observations,). Scales must be positive.\n Empty observations recover the isotropic zero-mean prior.\n \"\"\"\n # Key APIs to use: `jnp.asarray`, `contract`, `or`, `be`, `np.isfinite`\n # Step 1: Create device-backed JAX array `(X, y)`.\n # Step 2: Guard input contract (`X.ndim != 2 or y.ndim != 1 or X.shape[0] != y.shape[0] or (X.shape[1] == 0)`) and fail fast if violated.\n # Step 3: Guard input contract (`not np.isfinite(np.asarray(X)).all() or not np.isfinite(np.asarray(y)).all()`) and fail fast if violated.\n # Step 4: Guard input contract (`not np.isfinite([noise_scale, prior_scale]).all() or min(noise_scale, prior_scale) <= 0`) and fail fast if violated.\n # Step 5: Construct an identity matrix `precision`.\n # Step 6: Perform matrix contraction / projection to compute `mean`.\n raise NotImplementedError(\"Derive and solve the posterior precision system\")\n\ndef log_joint(weights, X, y, noise_scale=1., prior_scale=2.):\n \"\"\"Stage 1: scalar normalized Gaussian prior + independent log likelihood.\n Must support JAX differentiation with respect to the weights.\n \"\"\"\n # Key APIs to use: `jnp.sum`, `jnp.log`, `return`\n # Step 1: Perform matrix contraction / projection to compute `residual`.\n # Step 2: Evaluate `(n, d)` from the current inputs and state.\n # Step 3: Return `-0.5 * jnp.sum(residual ** 2) / noise_scale ** 2 - n * jnp.log(noise_scale) - 0.5 * jnp.sum(weights ** 2) / prior_scale ** 2 - d * jnp.log(prior_scale) - 0.5 * (n + d) * jnp.log(2 * jnp.pi)` to the caller.\n raise NotImplementedError(\"Keep scale-dependent normalization terms\")", "solutionSnippet": "# Function `fit` reference implementation:\ndef fit(X, y, noise_scale=1., prior_scale=2.):\n \"\"\"Zero-mean isotropic Gaussian prior; known positive observation-noise scale.\"\"\"\n # Create device-backed JAX array `(X, y)`.\n X, y = jnp.asarray(X, dtype=jnp.float32), jnp.asarray(y, dtype=jnp.float32)\n # Guard input contract (`X.ndim != 2 or y.ndim != 1 or X.shape[0] != y.shape[0] or (X.shape[1] == 0)`) and fail fast if violated.\n if X.ndim != 2 or y.ndim != 1 or X.shape[0] != y.shape[0] or X.shape[1] == 0:\n raise ValueError(\"X must be (observations, features), y must be (observations,)\")\n # Guard input contract (`not np.isfinite(np.asarray(X)).all() or not np.isfinite(np.asarray(y)).all()`) and fail fast if violated.\n if not np.isfinite(np.asarray(X)).all() or not np.isfinite(np.asarray(y)).all():\n raise ValueError(\"observations must be finite\")\n # Guard input contract (`not np.isfinite([noise_scale, prior_scale]).all() or min(noise_scale, prior_scale) <= 0`) and fail fast if violated.\n if not np.isfinite([noise_scale, prior_scale]).all() or min(noise_scale, prior_scale) <= 0:\n raise ValueError(\"scales must be positive and finite\")\n # Initialize array `precision` with explicit values and shape.\n precision = jnp.eye(X.shape[1])/prior_scale**2 + X.T@X/noise_scale**2\n # Perform matrix contraction / projection to compute `mean`.\n mean = jnp.linalg.solve(precision, X.T@y/noise_scale**2)\n # Construct an identity matrix `covariance`.\n covariance = jnp.linalg.solve(precision, jnp.eye(X.shape[1]))\n # Return `(mean, covariance)` to the caller.\n return mean, covariance\n\n# Function `log_joint` reference implementation:\ndef log_joint(weights, X, y, noise_scale=1., prior_scale=2.):\n \"\"\"Normalized prior and independent Gaussian likelihood; weights is one vector.\"\"\"\n # Perform matrix contraction / projection to compute `residual`.\n residual = y-X@weights\n # Compute `n, d` as `X.shape`.\n n, d = X.shape\n # Return `-0.5 * jnp.sum(residual ** 2) / noise_scale ** 2 - n * jnp.log(noise_scale) - 0.5 * jnp.sum(weights ** 2) / prior_scale ** 2 - d * jnp.log(prior_scale) - 0.5 * (n + d) * jnp.log(2 * jnp.pi)` to the caller.\n return (-.5*jnp.sum(residual**2)/noise_scale**2-n*jnp.log(noise_scale)\n -.5*jnp.sum(weights**2)/prior_scale**2-d*jnp.log(prior_scale)\n -.5*(n+d)*jnp.log(2*jnp.pi))" }, { "id": "2", "title": "Sampler and diagnostic audit", "command": "python3 projects/bayesian-regression/tests/check.py --stage 2 --implementation projects/bayesian-regression/my_model.py", "evidence": "Four traces, seeded replay, exact moment comparison, signed energy errors and classical split R-hat", "expected": "PASS stage 2: two-seed moments, replay, changed keys, R-hat oracle, stuck and unstable chains", "diagnosis": "Inspect leapfrog half steps, retained rejected states and independent key ownership.", "codeGuide": "In `projects/bayesian-regression/my_model.py`, implement `sample(...)`, `diagnose(...)` using `map`, `contract`, `or`, `np.isfinite`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def sample(key, mean, covariance, starts, step_size=.15, leapfrog_steps=9,\n warmup=300, draws=1800):\n \"\"\"Stage 2: HMC with a fixed identity mass matrix, independent chain keys.\n Return dict: chains (chains, draws, parameters), accepted (chains, draws),\n energy_error (chains, draws). Discard warmup before returning.\n Keep rejected positions. Reject nonfinite energy errors.\n \"\"\"\n # Key APIs to use: `map`, `contract`, `or`, `np.isfinite`, `np.asarray`\n # Step 1: Create device-backed JAX array `(mean, covariance, starts)`.\n # Step 2: Evaluate `d` from the current inputs and state.\n # Step 3: Guard input contract (`mean.ndim != 1 or covariance.shape != (d, d) or starts.ndim != 2 or (starts.shape[1] != d)`) and fail fast if violated.\n # Step 4: Guard input contract (`draws < 4 or warmup < 0 or leapfrog_steps < 1 or (not np.isfinite(step_size)) or (step_size <= 0)`) and fail fast if violated.\n # Step 5: Convert `host_cov` to a host NumPy array for inspection or verification.\n # Step 6: Guard input contract (`not np.isfinite(host_cov).all() or not np.allclose(host_cov, host_cov.T, atol=1e-06) or np.linalg.eigvalsh(host_cov).min() <= 0`) and fail fast if violated.\n raise NotImplementedError(\"Implement leapfrog, log acceptance and state ownership\")\n\ndef diagnose(chains):\n \"\"\"Stage 2: return mean and classical_split_rhat per parameter.\n Require multiple finite chains\n reject zero within-chain variance.\n This is classical split R-hat, not rank-normalized R-hat or ESS.\n \"\"\"\n # Key APIs to use: `np.asarray`, `contract`, `or`, `np.isfinite`, `all`\n # Step 1: Convert `values` to a host NumPy array for inspection or verification.\n # Step 2: Guard input contract (`values.ndim != 3 or values.shape[0] < 2 or values.shape[1] < 4 or (not np.isfinite(values).all())`) and fail fast if violated.\n # Step 3: Evaluate `half` from the current inputs and state.\n # Step 4: Combine or mask array elements to form `split`.\n # Step 5: Reduce across the target axis to summarize `within`.\n # Step 6: Guard input contract (`np.any(within <= 0)`) and fail fast if violated.\n raise NotImplementedError(\"Split chains and compare between/within variance\")", "solutionSnippet": "# Function `sample` reference implementation:\ndef sample(key, mean, covariance, starts, step_size=.15, leapfrog_steps=9,\n warmup=300, draws=1800):\n \"\"\"Return retained chains, acceptance flags and signed energy errors.\n Fixed warmup discard\n no adaptive step size, NUTS or mass-matrix adaptation.\n \"\"\"\n # Create device-backed JAX array `(mean, covariance, starts)`.\n mean, covariance, starts = map(jnp.asarray, (mean, covariance, starts))\n # Compute `d` as `mean.size`.\n d = mean.size\n # Guard input contract (`mean.ndim != 1 or covariance.shape != (d, d) or starts.ndim != 2 or (starts.shape[1] != d)`) and fail fast if violated.\n if mean.ndim != 1 or covariance.shape != (d,d) or starts.ndim != 2 or starts.shape[1] != d:\n raise ValueError(\"mean, covariance and chain-start shapes disagree\")\n # Guard input contract (`draws < 4 or warmup < 0 or leapfrog_steps < 1 or (not np.isfinite(step_size)) or (step_size <= 0)`) and fail fast if violated.\n if draws < 4 or warmup < 0 or leapfrog_steps < 1 or not np.isfinite(step_size) or step_size <= 0:\n raise ValueError(\"invalid sampler configuration\")\n # Convert `host_cov` to a host NumPy array for inspection or verification.\n host_cov = np.asarray(covariance)\n # Guard input contract (`not np.isfinite(host_cov).all() or not np.allclose(host_cov, host_cov.T, atol=1e-06) or np.linalg.eigvalsh(host_cov).min() <= 0`) and fail fast if violated.\n if (not np.isfinite(host_cov).all()\n or not np.allclose(host_cov, host_cov.T, atol=1e-6)\n or np.linalg.eigvalsh(host_cov).min() <= 0):\n raise ValueError(\"covariance must be finite, symmetric and positive definite\")\n # Guard input contract (`not np.isfinite(np.asarray(mean)).all() or not np.isfinite(np.asarray(starts)).all()`) and fail fast if violated.\n if not np.isfinite(np.asarray(mean)).all() or not np.isfinite(np.asarray(starts)).all():\n raise ValueError(\"mean and starts must be finite\")\n # Initialize array `precision` with explicit values and shape.\n precision = jnp.linalg.solve(covariance, jnp.eye(d))\n # Function `energy(q)` implementing this stage's computation:\n def energy(q):\n # Compute `delta` as `q-mean`.\n delta = q-mean\n # Return `0.5 * delta @ precision @ delta` to the caller.\n return .5*delta@precision@delta\n # Differentiate the objective to obtain gradients `force`.\n force = jax.grad(energy)\n\n # Function `one_chain(chain_key, start)` implementing this stage's computation:\n def one_chain(chain_key, start):\n # Function `transition(state, _)` implementing this stage's computation:\n def transition(state, _):\n # Compute `q, chain_key` as `state`.\n q, chain_key = state\n # Split the PRNG key deterministically into independent subkeys (`(chain_key, momentum_key, accept_key)`).\n chain_key, momentum_key, accept_key = jax.random.split(chain_key,3)\n # Draw pseudorandom samples for `initial_p` using the explicit RNG state.\n initial_p = jax.random.normal(momentum_key,q.shape)\n # Compute `p` as `initial_p-.5*step_size*force(q)`.\n p = initial_p-.5*step_size*force(q)\n # Function `leap(i, state)` implementing this stage's computation:\n def leap(i,state):\n # Compute `position,momentum` as `state`.\n position,momentum = state\n # Compute `position` as `position+step_size*momentum`.\n position = position+step_size*momentum\n # Combine or mask array elements to form `momentum`.\n momentum = momentum-jnp.where(in\",X,covariance,X)\n # Return `{'mean': X @ mean, 'latent_variance': latent, 'observation_variance': latent + noise_scale ** 2}` to the caller.\n return {\"mean\":X@mean,\"latent_variance\":latent,\"observation_variance\":latent+noise_scale**2}" } ], "rubric": [ "State a prior, likelihood and known-noise assumption", "Derive the conjugate posterior and verify independent numerical references", "Check gradients before tuning an HMC sampler", "Record replay, changed-seed behavior and retained rejected states", "Explain classical split R-hat and its limitations", "Separate latent uncertainty from observation noise", "Diagnose deliberate model mismatch and avoid real-data calibration claims" ], "workspaceFile": "projects/bayesian-regression/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/bayesian-regression/starter/model.py projects/bayesian-regression/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/bayesian-regression/starter/model.py projects/bayesian-regression/my_model.py" } ], "referenceCommand": "python3 projects/bayesian-regression/tests/check.py --stage all --implementation solution", "setupNote": "Extract the ZIP into a fresh folder. Its top folder contains requirements-cpu.txt and projects/bayesian-regression; run commands from that top folder in the activated environment. Copy the starter scaffold `projects/bayesian-regression/starter/model.py` to `projects/bayesian-regression/my_model.py`, implement the stage functions in `projects/bayesian-regression/my_model.py`, and verify each stage with `python3 projects/bayesian-regression/tests/check.py --implementation projects/bayesian-regression/my_model.py --stage 1` (through `--stage 3`).", "assessmentId": "probability" }, { "schemaVersion": 1, "id": "derivative-audit", "title": "Audit derivatives and compile a tiny transformation", "pathwayId": "internals", "status": "authored", "hardware": "CPU float64", "source": "projects/derivative-audit/README.md", "requiredLessonIds": [ "internals-01", "internals-02", "internals-03", "internals-04" ], "stages": [ { "id": "1", "title": "Directional products and exact curvature", "command": "python3 projects/derivative-audit/tests/check.py --implementation projects/derivative-audit/my_model.py --stage 1", "evidence": "Three independent Jacobian/Hessian comparisons, adjoint identity and shape contract", "expected": "PASS stage 1: three independent Jacobian/Hessian references, adjoint identity and shape rejection", "diagnosis": "If the HVP disagrees with J-transpose-J times the direction, include the residual-weighted second derivative term.", "codeGuide": "In `projects/derivative-audit/my_model.py`, implement `analyze(...)` using `contract`, `shape`, `mapping`, `jnp.array`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def analyze(x,direction,weighting):\n \"\"\"For f(a,b)=(ab, sin(a)+b*b), return value, jvp, vjp, hvp.\n\n Return a dictionary with those keys. hvp belongs to 0.5*sum(f(x)**2).\n Reject any point/direction/weighting not shaped (2,).\n \"\"\"\n # Key APIs to use: `contract`, `shape`, `mapping`, `jnp.array`, `jnp.sin`\n # Step 1: Guard input contract (`x.shape != (2,) or direction.shape != (2,) or weighting.shape != (2,)`) and fail fast if violated.\n # Step 2: Function `mapping(z)` implementing this stage's computation:\n # Step 3: Compute forward-mode Jacobian-vector product (`(value, jvp)`).\n # Step 4: Evaluate primal output and reverse-mode pullback function (`vjp`).\n # Step 5: Evaluate `objective` from the current inputs and state.\n # Step 6: Differentiate the objective to obtain gradients `hvp`.\n raise NotImplementedError('Stage 1: forward, reverse and curvature products')", "solutionSnippet": "# Function `analyze` reference implementation:\ndef analyze(x,direction,weighting):\n # Guard input contract (`x.shape != (2,) or direction.shape != (2,) or weighting.shape != (2,)`) and fail fast if violated.\n if x.shape!=(2,) or direction.shape!=(2,) or weighting.shape!=(2,):\n raise ValueError(\"point, direction and weighting must each have shape (2,)\")\n # Function `mapping(z)` implementing this stage's computation:\n def mapping(z):\n # Compute `a,b` as `z`.\n a,b=z\n # Return `jnp.array([a * b, jnp.sin(a) + b * b])` to the caller.\n return jnp.array([a*b,jnp.sin(a)+b*b])\n # Compute exact directional derivative / Jacobian / Hessian (`(value, jvp)`).\n value,jvp=jax.jvp(mapping,(x,),(direction,))\n # Compute exact directional derivative / Jacobian / Hessian (`vjp`).\n vjp=jax.vjp(mapping,x)[1](weighting)[0]\n # Compute `objective` as `lambda z:0.5*jnp.vdot(mapping(z),mapping(z))`.\n objective=lambda z:0.5*jnp.vdot(mapping(z),mapping(z))\n # Differentiate the objective to obtain gradients `hvp`.\n hvp=jax.jvp(jax.grad(objective),(x,),(direction,))[1]\n # Return `{'value': value, 'jvp': jvp, 'vjp': vjp, 'hvp': hvp}` to the caller.\n return {\"value\":value,\"jvp\":jvp,\"vjp\":vjp,\"hvp\":hvp}" }, { "id": "2", "title": "Exact custom derivative contracts", "command": "python3 projects/derivative-audit/tests/check.py --implementation projects/derivative-audit/my_model.py --stage 2", "evidence": "Stable extreme values, analytic slopes/Hessians, cotangent linearity and transformation-boundary evidence", "expected": "PASS stage 2: stable exact custom primals, gradients, curvature, cotangent linearity, common shifts and custom-VJP boundary", "diagnosis": "If first derivatives pass but second derivatives fail, inspect detached derivative coefficients or saved residuals.", "codeGuide": "In `projects/derivative-audit/my_model.py`, implement `stable_softplus(...)`, `stable_logsumexp(...)` using `jnp.logaddexp`, `special.logsumexp`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def stable_softplus(x):\n \"\"\"Stable scalar softplus, with an exact registered JVP and higher derivatives.\"\"\"\n # Key APIs to use: `jnp.logaddexp`\n # Step 1: Return `jnp.logaddexp(0.0, x)` to the caller.\n raise NotImplementedError('Stage 2: stable primal plus exact custom JVP')\n\ndef stable_logsumexp(x):\n \"\"\"Stable vector logsumexp, with exact registered forward/backward rules.\"\"\"\n # Key APIs to use: `special.logsumexp`\n # Step 1: Return `jax.scipy.special.logsumexp(x)` to the caller.\n raise NotImplementedError('Stage 2: stable primal plus exact custom VJP')", "solutionSnippet": "# Function `stable_softplus` reference implementation:\ndef stable_softplus(x):\n # Return `jnp.logaddexp(0.0, x)` to the caller.\n return jnp.logaddexp(0.0,x)\n\n# Function `stable_logsumexp` reference implementation:\ndef stable_logsumexp(x):\n # Return `jax.scipy.special.logsumexp(x)` to the caller.\n return jax.scipy.special.logsumexp(x)" }, { "id": "3", "title": "A compiled primitive-level transformation", "command": "python3 projects/derivative-audit/tests/check.py --implementation projects/derivative-audit/my_model.py --stage 3", "evidence": "Explicit rules, random-shape expression oracles, captured constants, multiple inputs/outputs and unsupported-operation rejection", "expected": "PASS stage 3: random shapes, independent expression oracles, constants, multiple inputs/outputs, compiled changed values and explicit unsupported/signature rejection", "diagnosis": "If the primal is correct but the tangent is not, audit multiplication and constant tangents before changing compiler settings.", "codeGuide": "In `projects/derivative-audit/my_model.py`, implement `tiny_jvp(...)` using `contract`, `NotImplementedError`, `put`, `zip`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def tiny_jvp(closed,primals,tangents):\n \"\"\"Return flat tuples (primal_outputs, tangent_outputs) by interpreting a jaxpr.\n\n Support add, mul, neg, sin and reduce_sum using explicit tangent rules;\n initialize captured/literal constants with zero tangents. Reject effects,\n unsupported/multi-result primitives, nonfloating inputs, explicit reduction\n placement and input/tangent shape or dtype mismatches. Do not delegate the\n derivative calculation to jax.jvp, grad, jacfwd or jacrev.\n \"\"\"\n # Key APIs to use: `contract`, `NotImplementedError`, `put`, `zip`, `jnp.zeros_like`\n # Step 1: Evaluate `program` from the current inputs and state.\n # Step 2: Guard input contract (`program.effects`) and fail fast if violated.\n # Step 3: Guard input contract (`len(primals) != len(program.invars) or len(tangents) != len(primals)`) and fail fast if violated.\n # Step 4: Evaluate `values` from the current inputs and state.\n # Step 5: Evaluate `directions` from the current inputs and state.\n # Step 6: Function `put(var, value, tangent)` implementing this stage's computation:\n raise NotImplementedError('Stage 3: actual primitive-by-primitive transformation')", "solutionSnippet": "# Function `tiny_jvp` reference implementation:\ndef tiny_jvp(closed,primals,tangents):\n \"\"\"Interpret a pure flat jaxpr with floating inputs and explicit tangent rules.\"\"\"\n # Compute `program` as `closed.jaxpr`.\n program=closed.jaxpr\n # Guard input contract (`program.effects`) and fail fast if violated.\n if program.effects:\n raise NotImplementedError(\"effects are outside this interpreter\")\n # Guard input contract (`len(primals) != len(program.invars) or len(tangents) != len(primals)`) and fail fast if violated.\n if len(primals)!=len(program.invars) or len(tangents)!=len(primals):\n raise ValueError(\"one primal and tangent per input variable required\")\n # Construct dictionary `values` with the structured fields for this stage.\n values={}\n directions={}\n # Function `put(var, value, tangent)` implementing this stage's computation:\n def put(var,value,tangent):\n # Compute `values[var]` as `value`.\n values[var]=value\n directions[var]=tangent\n # Loop over `(var, constant)` in `zip(program.constvars, closed.consts)`:\n for var,constant in zip(program.constvars,closed.consts):\n # Allocate initialized array `` with the specified shape and dtype.\n put(var,constant,jnp.zeros_like(constant))\n # Loop over `(var, value, tangent)` in `zip(program.invars, primals, tangents)`:\n for var,value,tangent in zip(program.invars,primals,tangents):\n # Guard input contract (`value.shape != var.aval.shape or value.dtype != var.aval.dtype`) and fail fast if violated.\n if value.shape!=var.aval.shape or value.dtype!=var.aval.dtype:\n raise ValueError(\"primal shape/dtype differs from traced signature\")\n # Guard input contract (`tangent.shape != value.shape or tangent.dtype != value.dtype`) and fail fast if violated.\n if tangent.shape!=value.shape or tangent.dtype!=value.dtype:\n raise ValueError(\"tangent shape/dtype must match its primal\")\n # Guard input contract (`not jnp.issubdtype(value.dtype, jnp.floating)`) and fail fast if violated.\n if not jnp.issubdtype(value.dtype,jnp.floating):\n raise ValueError(\"this teaching interpreter requires real floating inputs\")\n # Run `put` to perform the next check or state transition.\n put(var,value,tangent)\n # Function `read(atom)` implementing this stage's computation:\n def read(atom):\n # Branch on condition `isinstance(atom, core.Literal)`:\n if isinstance(atom,core.Literal):\n return atom.val,jnp.zeros_like(atom.val)\n # Return `(values[atom], directions[atom])` to the caller.\n return values[atom],directions[atom]\n # Loop over `equation` in `program.eqns`:\n for equation in program.eqns:\n # Compute `name` as `equation.primitive.name`.\n name=equation.primitive.name\n # Guard input contract (`name not in {'add', 'mul', 'neg', 'sin', 'reduce_sum'}`) and fail fast if violated.\n if name not in {\"add\",\"mul\",\"neg\",\"sin\",\"reduce_sum\"}:\n raise NotImplementedError(\"unsupported primitive: \"+name)\n # Guard input contract (`len(equation.outvars) != 1`) and fail fast if violated.\n if len(equation.outvars)!=1:\n raise NotImplementedError(\"multiple-result primitives are outside this interpreter\")\n # Initialize list `operands` for the stage values.\n operands=[read(atom) for atom in equation.invars]\n # Compute `a,da` as `operands[0]`.\n a,da=operands[0]\n # Branch on condition `name == 'add'`:\n if name==\"add\":\n b,db=operands[1]\n result,tangent=a+b,da+db\n elif name==\"mul\":\n b,db=operands[1]\n result,tangent=a*b,da*b+a*db\n elif name==\"neg\":result,tangent=-a,-da\n elif name==\"sin\":result,tangent=jnp.sin(a),jnp.cos(a)*da\n else:\n if equation.params.get(\"out_sharding\") is not None:\n raise NotImplementedError(\"explicit reduction placement is outside this interpreter\")\n axes=equation.params[\"axes\"]\n result,tangent=jnp.sum(a,axis=axes),jnp.sum(da,axis=axes)\n # Run `put` to perform the next check or state transition.\n put(equation.outvars[0],result,tangent)\n # Initialize list `output` for the stage values.\n output=[read(var) for var in program.outvars]\n # Return `(tuple((v for v, _ in output)), tuple((t for _, t in output)))` to the caller.\n return tuple(v for v,_ in output),tuple(t for _,t in output)" } ], "rubric": [ "Explain input directions and output cotangents using independent Jacobians", "Verify an exact Hessian-vector product rather than substituting Gauss–Newton", "Defend exact custom JVP/VJP rules, stability, seed linearity and higher derivatives", "Implement explicit primitive tangent rules with captured and literal constants", "Reject unsupported primitives and shape/dtype violations", "Verify interpreted and compiled transformed outputs on changed values and shapes", "Explain limits of the supported subset and distinguish IR inspection from performance measurement" ], "workspaceFile": "projects/derivative-audit/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/derivative-audit/starter/model.py projects/derivative-audit/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/derivative-audit/starter/model.py projects/derivative-audit/my_model.py" } ], "referenceCommand": "python3 projects/derivative-audit/tests/check.py --implementation solution --stage all", "setupNote": "Extract the ZIP, install requirements-cpu.txt in your course environment, and run commands from the extracted top folder. The scaffold enables CPU float64 examples; no compiler or accelerator installation is required. Copy the starter scaffold `projects/derivative-audit/starter/model.py` to `projects/derivative-audit/my_model.py`, implement the stage functions in `projects/derivative-audit/my_model.py`, and verify each stage with `python3 projects/derivative-audit/tests/check.py --implementation projects/derivative-audit/my_model.py --stage 1` (through `--stage 3`).", "assessmentId": "internals" }, { "schemaVersion": 1, "id": "deployment-audit", "title": "Release a trained computation with a deployment audit", "pathwayId": "ship", "status": "authored", "hardware": "CPU; device and network validation remain separate", "source": "projects/deployment-audit/README.md", "requiredLessonIds": [ "deployment-01", "deployment-03", "deployment-05", "deployment-06", "deployment-02", "deployment-04" ], "stages": [ { "id": "1", "title": "Trained artifact and adaptation provenance", "command": "python3 projects/deployment-audit/tests/check.py --implementation projects/deployment-audit/my_model.py --stage 1", "evidence": "Stable objective oracle, actual pretraining, saved/reloaded base and changed-task update", "expected": "PASS stage 1: independent objective gradient, genuine pretraining, checkpoint reload and changed-task adaptation", "diagnosis": "Check objective reduction, target shape and checkpoint identity before altering the learning rate.", "codeGuide": "In `projects/deployment-audit/my_model.py`, implement `objective(...)`, `train(...)` using `contraction`, `jnp.mean`, `jnp.logaddexp`, `map`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def objective(weights,X,targets):\n \"\"\"Stage 1: stable mean binary cross-entropy, supporting soft targets and autodiff.\"\"\"\n # Key APIs to use: `contraction`, `jnp.mean`, `jnp.logaddexp`\n # Step 1: Perform matrix contraction / projection to compute `scores`.\n # Step 2: Return `jnp.mean(jnp.logaddexp(0.0, scores) - targets * scores)` to the caller.\n raise NotImplementedError(\"Define the objective\")\n\ndef train(initial,X,targets,steps=250,rate=.15):\n \"\"\"Stage 1: return trained weights and pre-update loss history; validate inputs.\"\"\"\n # Key APIs to use: `map`, `contract`, `X`, `targets`, `weights`\n # Step 1: Create device-backed JAX array `(X, targets, initial)`.\n # Step 2: Guard input contract (`X.ndim != 2 or targets.shape != (X.shape[0],) or initial.shape != (X.shape[1],)`) and fail fast if violated.\n # Step 3: Guard input contract (`len(X) == 0 or not np.isfinite(np.asarray(X)).all() or (not np.isfinite(np.asarray(targets)).all()) or (not np.isfinite(np.asarray(initial)).all())`) and fail fast if violated.\n # Step 4: Guard input contract (`np.any((np.asarray(targets) < 0) | (np.asarray(targets) > 1))`) and fail fast if violated.\n # Step 5: Guard input contract (`steps < 0 or not np.isfinite(rate) or rate <= 0`) and fail fast if violated.\n # Step 6: Function `update(w, _)` implementing this stage's computation:\n raise NotImplementedError(\"Adapt explicit initial weights with SGD\")", "solutionSnippet": "# Function `objective` reference implementation:\ndef objective(weights, X, targets):\n # Perform matrix / vector contraction (`@`) to compute `scores`.\n scores=X@weights\n # Return `jnp.mean(jnp.logaddexp(0.0, scores) - targets * scores)` to the caller.\n return jnp.mean(jnp.logaddexp(0.,scores)-targets*scores)\n\n# Function `train` reference implementation:\ndef train(initial, X, targets, steps=250, rate=.15):\n # Create device-backed JAX array `(X, targets, initial)`.\n X,targets,initial=map(jnp.asarray,(X,targets,initial))\n # Guard input contract (`X.ndim != 2 or targets.shape != (X.shape[0],) or initial.shape != (X.shape[1],)`) and fail fast if violated.\n if X.ndim!=2 or targets.shape!=(X.shape[0],) or initial.shape!=(X.shape[1],):\n raise ValueError(\"expected X (rows,features), targets (rows,), weights (features,)\")\n # Guard input contract (`len(X) == 0 or not np.isfinite(np.asarray(X)).all() or (not np.isfinite(np.asarray(targets)).all()) or (not np.isfinite(np.asarray(initial)).all())`) and fail fast if violated.\n if len(X)==0 or not np.isfinite(np.asarray(X)).all() or not np.isfinite(np.asarray(targets)).all() or not np.isfinite(np.asarray(initial)).all():\n raise ValueError(\"training data must be finite and nonempty\")\n # Guard input contract (`np.any((np.asarray(targets) < 0) | (np.asarray(targets) > 1))`) and fail fast if violated.\n if np.any((np.asarray(targets)<0)|(np.asarray(targets)>1)):\n raise ValueError(\"binary/soft targets must lie between zero and one\")\n # Guard input contract (`steps < 0 or not np.isfinite(rate) or rate <= 0`) and fail fast if violated.\n if steps<0 or not np.isfinite(rate) or rate<=0:raise ValueError(\"invalid optimizer configuration\")\n # Define `update(w, _)` to evaluate the objective and its automatic derivatives:\n def update(w,_):\n # Differentiate the objective to obtain `(loss, grad)` via automatic differentiation.\n loss,grad=jax.value_and_grad(objective)(w,X,targets)\n # Return `(w - rate * grad, loss)` to the caller.\n return w-rate*grad,loss\n # Return `jax.lax.scan(update, initial, None, length=steps)` to the caller.\n return jax.lax.scan(update,initial,None,length=steps)" }, { "id": "2", "title": "Export and precision contract", "command": "python3 projects/deployment-audit/tests/check.py --implementation projects/deployment-audit/my_model.py --stage 2", "evidence": "Six real serialized artifacts, independent output and integer arithmetic checks, calibration hashes and corruption rejection", "expected": "PASS stage 2: six exported round trips, independent precision arithmetic, calibration provenance and corruption rejection", "diagnosis": "Inspect weight axes, scale granularity, activation calibration and the actual restored artifact.", "codeGuide": "In `projects/deployment-audit/my_model.py`, implement `quantize(...)`, `prepare(...)`, `load(...)` using `a`, `np.asarray`, `contract`, `np.isfinite`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def quantize(weights,bits=8):\n \"\"\"Stage 2: matrix -> signed int8-container codes, per-output scales; bits 4 or 8.\"\"\"\n # Key APIs to use: `a`, `np.asarray`, `contract`, `np.isfinite`, `all`\n # Step 1: Convert `weights` to a host NumPy array for inspection or verification.\n # Step 2: Guard input contract (`weights.ndim != 2 or not np.isfinite(weights).all() or bits not in (4, 8)`) and fail fast if violated.\n # Step 3: Evaluate `limit` from the current inputs and state.\n # Step 4: Reduce across the target axis to summarize `maximum`.\n # Step 5: Cast or evaluate `scales` in explicit floating-point precision.\n # Step 6: Combine or mask array elements to form `codes`.\n raise NotImplementedError(\"Implement symmetric rounding and zero-column behavior\")\n\ndef prepare(directory,weights,calibration,provenance):\n \"\"\"Stage 2: write weights.npy, manifest.json and six serialized JAX exports.\n Policies fp32/w8a32/w8a8\n batches 1 and 8. W8A8 has int32 accumulation.\n Preserve calibration hash, artifact hashes, precision fields and provenance.\n \"\"\"\n # Key APIs to use: `disk`, `Path`, `directory.mkdir`, `np.asarray`, `contract`\n # Step 1: Read or serialize artifact data on disk (`directory`).\n # Step 2: Execute the next step of the computation.\n # Step 3: Convert `weights` to a host NumPy array for inspection or verification.\n # Step 4: Convert `calibration` to a host NumPy array for inspection or verification.\n # Step 5: Guard input contract (`weights.ndim != 1 or calibration.ndim != 2 or calibration.shape[1] != weights.size or (len(calibration) == 0)`) and fail fast if violated.\n # Step 6: Guard input contract (`not np.isfinite(calibration).all() or not np.isfinite(weights).all()`) and fail fast if violated.\n raise NotImplementedError(\"Package versioned inference artifacts\")\n\ndef load(directory):\n \"\"\"Stage 2: verify hashes and return (manifest, restored exports keyed policy:batch).\"\"\"\n # Key APIs to use: `disk`, `Path`, `json.loads`, `read_text`, `contract`\n # Step 1: Read or serialize artifact data on disk (`directory`).\n # Step 2: Read or serialize artifact data on disk (`manifest`).\n # Step 3: Guard input contract (`hashlib.sha256((directory / 'weights.npy').read_bytes()).hexdigest() != manifest['weights_sha256']`) and fail fast if violated.\n # Step 4: Evaluate `restored` from the current inputs and state.\n # Step 5: Loop over `(key, item)` in `manifest['artifacts'].items()`:\n # Step 6: Inside block: Evaluate `data` from the current inputs and state.\n raise NotImplementedError(\"Read and verify actual serialized bytes\")", "solutionSnippet": "# Function `quantize` reference implementation:\ndef quantize(weights,bits=8):\n \"\"\"Per-output symmetric codes for a (features, outputs) matrix.\"\"\"\n # Convert `weights` to a host NumPy array for inspection or verification.\n weights=np.asarray(weights,dtype=np.float32)\n # Guard input contract (`weights.ndim != 2 or not np.isfinite(weights).all() or bits not in (4, 8)`) and fail fast if violated.\n if weights.ndim!=2 or not np.isfinite(weights).all() or bits not in (4,8):\n raise ValueError(\"finite matrix and signed 4/8-bit policy required\")\n # Compute `limit` as `2**(bits-1)-1`.\n limit=2**(bits-1)-1\n # Reduce along axis=0 to compute `maximum`.\n maximum=np.max(np.abs(weights),axis=0,keepdims=True)\n # Cast or evaluate `scales` in explicit floating-point precision.\n scales=np.where(maximum==0,1.,maximum/limit).astype(np.float32)\n # Combine or mask array elements to form `codes`.\n codes=np.clip(np.rint(weights/scales),-limit,limit).astype(np.int8)\n # Return `(codes, scales)` to the caller.\n return codes,scales\n\n# Function `prepare` reference implementation:\ndef prepare(directory,weights,calibration,provenance):\n \"\"\"Write trained weights and six actual serialized inference computations.\"\"\"\n # Compute `directory` as `Path(directory)`.\n directory=Path(directory)\n directory.mkdir(parents=True,exist_ok=True)\n # Convert `weights` to a host NumPy array for inspection or verification.\n # Convert `calibration` to a host NumPy array for inspection or verification.\n weights=np.asarray(weights,dtype=np.float32)\n calibration=np.asarray(calibration,dtype=np.float32)\n # Guard input contract (`weights.ndim != 1 or calibration.ndim != 2 or calibration.shape[1] != weights.size or (len(calibration) == 0)`) and fail fast if violated.\n if weights.ndim!=1 or calibration.ndim!=2 or calibration.shape[1]!=weights.size or len(calibration)==0:\n raise ValueError(\"calibration and weights must have matching feature axes\")\n # Guard input contract (`not np.isfinite(calibration).all() or not np.isfinite(weights).all()`) and fail fast if violated.\n if not np.isfinite(calibration).all() or not np.isfinite(weights).all():\n raise ValueError(\"finite calibration/weights required\")\n # Compute deterministic cryptographic digest `required` for provenance verification.\n required={\"source_data_sha256\",\"adaptation_data_sha256\",\"training_steps\",\"objective\",\"base_checkpoint_sha256\"}\n # Guard input contract (`not required <= provenance.keys()`) and fail fast if violated.\n if not required<=provenance.keys():raise ValueError(\"training and base-checkpoint provenance required\")\n # Run `quantize` to compute `(q, s)`.\n q,s=quantize(weights[:,None],8)\n # Cast or evaluate `activation_scale` in explicit floating-point precision.\n activation_scale=np.float32(max(float(np.max(np.abs(calibration)))/127,1e-8))\n # Run `np.save` to perform the next check or state transition.\n np.save(directory/\"weights.npy\",weights,allow_pickle=False)\n # Construct dictionary `policies` with the structured fields for this stage.\n policies={\n \"fp32\":dict(weight=\"float32\",activation=\"float32\",accumulator=\"float32\",output=\"float32\"),\n \"w8a32\":dict(weight=\"int8\",activation=\"float32\",accumulator=\"float32\",output=\"float32\"),\n \"w8a8\":dict(weight=\"int8\",activation=\"int8\",accumulator=\"int32\",output=\"float32\"),\n }\n # Function `make_infer(policy)` implementing this stage's computation:\n def make_infer(policy):\n # Branch on condition `policy == 'fp32'`:\n if policy==\"fp32\":\n return jax.jit(lambda x:x@jnp.asarray(weights))\n # Branch on condition `policy == 'w8a32'`:\n if policy==\"w8a32\":\n return jax.jit(lambda x:(x@(jnp.asarray(q,dtype=jnp.float32)*jnp.asarray(s))).reshape(-1))\n # Function `integer(x)` implementing this stage's computation:\n def integer(x):\n # Combine or mask array elements to form `qa`.\n qa=jnp.clip(jnp.rint(x/activation_scale),-127,127).astype(jnp.int8)\n # Create device-backed JAX array `accum`.\n accum=qa.astype(jnp.int32)@jnp.asarray(q,dtype=jnp.int32)\n # Return `(accum.astype(jnp.float32) * (activation_scale * jnp.asarray(s))).reshape(-1)` to the caller.\n return (accum.astype(jnp.float32)*(activation_scale*jnp.asarray(s))).reshape(-1)\n # Return `jax.jit(integer)` to the caller.\n return jax.jit(integer)\n # Construct dictionary `artifacts` with the structured fields for this stage.\n artifacts={}\n # Loop over `policy` in `policies`:\n for policy in policies:\n # Loop over `batch` in `(1, 8)`:\n for batch in (1,8):\n # Cast or evaluate `artifact` in explicit floating-point precision.\n artifact=export.export(make_infer(policy))(jax.ShapeDtypeStruct((batch,len(weights)),jnp.float32))\n # Compute `name` as `f\"{policy}-b{batch}.jaxexport\"`.\n name=f\"{policy}-b{batch}.jaxexport\"\n # Run `artifact.serialize` to compute `data`.\n # Compute `data` as `artifact.serialize()`.\n data=artifact.serialize()\n (directory/name).write_bytes(data)\n # Compute deterministic cryptographic digest `artifacts[f'{policy}:{batch}']` for provenance verification.\n artifacts[f\"{policy}:{batch}\"]={\"file\":name,\"sha256\":hashlib.sha256(data).hexdigest()}\n # Compute deterministic cryptographic digest `manifest` for provenance verification.\n manifest={\"schema_version\":1,\"features\":len(weights),\"batch_sizes\":[1,8],\"jax\":jax.__version__,\n \"platform\":\"cpu\",\"preprocessing\":\"identity float32 features; intercept is caller-supplied\",\n \"weights_sha256\":hashlib.sha256((directory/\"weights.npy\").read_bytes()).hexdigest(),\n \"calibration_sha256\":hashlib.sha256(calibration.tobytes()).hexdigest(),\n \"activation_scale\":float(activation_scale),\"weight_scales\":s.tolist(),\n \"policies\":policies,\"artifacts\":artifacts,\"provenance\":provenance,\n \"boundary\":\"serialized JAX CPU inference; integer kernel acceleration unclaimed\"}\n # Compute `(directory/\"manifest.json\").write_text(json.dumps(manifest,indent` as `2))`.\n (directory/\"manifest.json\").write_text(json.dumps(manifest,indent=2))\n # Return `manifest` to the caller.\n return manifest\n\n# Function `load` reference implementation:\ndef load(directory):\n # Read or serialize artifact data on disk (`directory`).\n directory=Path(directory)\n # Read or serialize artifact data on disk (`manifest`).\n manifest=json.loads((directory/\"manifest.json\").read_text())\n # Guard input contract (`hashlib.sha256((directory / 'weights.npy').read_bytes()).hexdigest() != manifest['weights_sha256']`) and fail fast if violated.\n if hashlib.sha256((directory/\"weights.npy\").read_bytes()).hexdigest()!=manifest[\"weights_sha256\"]:\n raise ValueError(\"checkpoint hash mismatch\")\n # Construct dictionary `restored` with the structured fields for this stage.\n restored={}\n # Iterate over `(key, item)` to step through the computation:\n for key,item in manifest[\"artifacts\"].items():\n # Evaluate the compound expression for `data`.\n data=(directory/item[\"file\"]).read_bytes()\n # Guard input contract (`hashlib.sha256(data).hexdigest() != item['sha256']`) and fail fast if violated.\n if hashlib.sha256(data).hexdigest()!=item[\"sha256\"]:raise ValueError(\"artifact hash mismatch\")\n # Run `export.deserialize` to compute `restored[key]`.\n restored[key]=export.deserialize(data)\n # Return `(manifest, restored)` to the caller.\n return manifest,restored" }, { "id": "3", "title": "Request timing and capacity evidence", "command": "python3 projects/deployment-audit/tests/check.py --implementation projects/deployment-audit/my_model.py --stage 3", "evidence": "Validated JSON requests, completed batch timings, clipping reports and independently checked queue assumptions", "expected": "PASS stage 3: real JSON request parity, timed batch requests, clipping, bad inputs and independent queue timelines", "diagnosis": "Separate compute completion, request overhead and simulated queue delay; preserve units.", "codeGuide": "In `projects/deployment-audit/my_model.py`, implement `request(...)`, `benchmark(...)`, `simulate(...)` using `contract`, `payload.encode`, `json.loads`, `np.asarray`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def request(payload,manifest,restored,policy=\"fp32\"):\n \"\"\"Stage 3: bounded JSON features -> JSON scores, policy and clipped_activations.\n Validate batch/feature axes, numeric finite inputs and policy before inference.\n \"\"\"\n # Key APIs to use: `contract`, `payload.encode`, `json.loads`, `np.asarray`, `raw.astype`\n # Step 1: Guard input contract (`not isinstance(payload, str) or len(payload.encode('utf8')) > 100000`) and fail fast if violated.\n # Step 2: Guard input contract (`x.ndim != 2 or x.shape[1] != manifest['features'] or x.shape[0] not in manifest['batch_sizes']`) and fail fast if violated.\n # Step 3: Guard input contract (`not np.isfinite(x).all() or policy not in manifest['policies']`) and fail fast if violated.\n # Step 4: Synchronize host execution until asynchronous device computation completes.\n # Step 5: Reduce across the target axis to summarize `clipped`.\n # Step 6: Return `json.dumps({'scores': np.asarray(result).tolist(), 'policy': policy, 'clipped_activations': clipped})` to the caller.\n raise NotImplementedError(\"Define the complete in-process request boundary\")\n\ndef benchmark(payload,manifest,restored,policy=\"fp32\",repeats=30):\n \"\"\"Stage 3: first_request_ms, samples_ms, p50_ms, p95_ms, examples_per_second.\n Include decode/validation/transfer/completion/encoding\n network_included=False.\n \"\"\"\n # Key APIs to use: `contract`, `time.perf_counter`, `request`, `values.append`, `disk`\n # Step 1: Guard input contract (`repeats < 2`) and fail fast if violated.\n # Step 2: Record execution timing or profiler trace in `began`.\n # Step 3: Execute the next step of the computation.\n # Step 4: Record execution timing or profiler trace in `first_ms`.\n # Step 5: Evaluate `values` from the current inputs and state.\n # Step 6: Repeat the update loop over `range(repeats)` steps:\n raise NotImplementedError(\"Measure completed requests and label boundaries\")\n\ndef simulate(arrival_ms,service_ms):\n \"\"\"Stage 3: simulated FCFS response times in milliseconds; not measured latency.\"\"\"\n # Key APIs to use: `np.asarray`, `contract`, `np.isfinite`, `all`, `np.any`\n # Step 1: Convert `arrivals` to a host NumPy array for inspection or verification.\n # Step 2: Guard input contract (`arrivals.ndim != 1 or not np.isfinite(arrivals).all() or np.any(arrivals < 0) or np.any(np.diff(arrivals) < 0)`) and fail fast if violated.\n # Step 3: Guard input contract (`not np.isfinite(service_ms) or service_ms <= 0`) and fail fast if violated.\n # Step 4: Evaluate `ready` from the current inputs and state.\n # Step 5: Evaluate `finish` from the current inputs and state.\n # Step 6: Loop over `arrival` in `arrivals`:\n raise NotImplementedError(\"Trace the declared one-worker queue\")", "solutionSnippet": "# Function `request` reference implementation:\ndef request(payload,manifest,restored,policy=\"fp32\"):\n \"\"\"In-process JSON service boundary: validation, transfer, inference, completion, encoding.\"\"\"\n # Guard input contract (`not isinstance(payload, str) or len(payload.encode('utf8')) > 100000`) and fail fast if violated.\n if not isinstance(payload,str) or len(payload.encode(\"utf8\"))>100_000:\n raise ValueError(\"payload must be a bounded JSON string\")\n # Run the boundary check and catch the expected exception:\n try:\n item=json.loads(payload)\n raw=np.asarray(item[\"features\"])\n if raw.dtype.kind not in \"iuf\":raise ValueError(\"features must be numeric\")\n x=raw.astype(np.float32)\n except (KeyError,TypeError,OverflowError,json.JSONDecodeError) as err:\n raise ValueError(\"malformed feature payload\") from err\n # Guard input contract (`x.ndim != 2 or x.shape[1] != manifest['features'] or x.shape[0] not in manifest['batch_sizes']`) and fail fast if violated.\n if x.ndim!=2 or x.shape[1]!=manifest[\"features\"] or x.shape[0] not in manifest[\"batch_sizes\"]:\n raise ValueError(\"unsupported request shape\")\n # Guard input contract (`not np.isfinite(x).all() or policy not in manifest['policies']`) and fail fast if violated.\n if not np.isfinite(x).all() or policy not in manifest[\"policies\"]:\n raise ValueError(\"nonfinite input or unsupported precision policy\")\n # Synchronize host execution until asynchronous device computation completes.\n result=restored[f\"{policy}:{len(x)}\"].call(jnp.asarray(x)).block_until_ready()\n # Aggregate array values to compute `clipped`.\n clipped=int(np.sum(np.abs(x)>127*manifest[\"activation_scale\"])) if policy==\"w8a8\" else 0\n # Return `json.dumps({'scores': np.asarray(result).tolist(), 'policy': policy, 'clipped_activations': clipped})` to the caller.\n return json.dumps({\"scores\":np.asarray(result).tolist(),\"policy\":policy,\"clipped_activations\":clipped})\n\n# Function `benchmark` reference implementation:\ndef benchmark(payload,manifest,restored,policy=\"fp32\",repeats=30):\n # Guard input contract (`repeats < 2`) and fail fast if violated.\n if repeats<2:raise ValueError(\"need multiple observations\")\n # Record execution timing or profiler trace in `began`.\n # Compute `began` as `time.perf_counter()`.\n began=time.perf_counter()\n request(payload,manifest,restored,policy)\n # Record execution timing or profiler trace in `first_ms`.\n first_ms=(time.perf_counter()-began)*1000\n # Initialize list `values` for the stage values.\n values=[]\n # Repeat the update loop over `range(repeats)` steps:\n for _ in range(repeats):\n # Record execution timing or profiler trace in `began`.\n # Compute `began` as `time.perf_counter()`.\n began=time.perf_counter()\n request(payload,manifest,restored,policy)\n # Record execution timing or profiler trace in ``.\n values.append((time.perf_counter()-began)*1000)\n # Read or serialize artifact data on disk (`batch`).\n batch=len(json.loads(payload)[\"features\"])\n # Return `{'first_request_ms': first_ms, 'samples_ms': values, 'batch': batch, 'p50_ms': float(np.percentile(values, 50)), 'p95_ms': float(np.percentile(values, 95)), 'examples_per_second': 1000 * batch / float(np.percentile(values, 50)), 'boundary': 'JSON decode + validate + transfer + restored call + wait + JSON encode', 'network_included': False, 'device': 'CPU'}` to the caller.\n return {\"first_request_ms\":first_ms,\"samples_ms\":values,\"batch\":batch,\n \"p50_ms\":float(np.percentile(values,50)),\"p95_ms\":float(np.percentile(values,95)),\n \"examples_per_second\":1000*batch/float(np.percentile(values,50)),\n \"boundary\":\"JSON decode + validate + transfer + restored call + wait + JSON encode\",\n \"network_included\":False,\"device\":\"CPU\"}\n\n# Function `simulate` reference implementation:\ndef simulate(arrival_ms,service_ms):\n # Convert `arrivals` to a host NumPy array for inspection or verification.\n arrivals=np.asarray(arrival_ms,dtype=float)\n # Guard input contract (`arrivals.ndim != 1 or not np.isfinite(arrivals).all() or np.any(arrivals < 0) or np.any(np.diff(arrivals) < 0)`) and fail fast if violated.\n if arrivals.ndim!=1 or not np.isfinite(arrivals).all() or np.any(arrivals<0) or np.any(np.diff(arrivals)<0):\n raise ValueError(\"ordered nonnegative finite arrival times required\")\n # Guard input contract (`not np.isfinite(service_ms) or service_ms <= 0`) and fail fast if violated.\n if not np.isfinite(service_ms) or service_ms<=0:raise ValueError(\"positive service milliseconds required\")\n # Compute `ready` as `0.`.\n ready=0.\n finish=[]\n # Iterate over `arrival` to step through the computation:\n for arrival in arrivals:\n # Run `max` to compute `ready`.\n # Compute `ready` as `max(ready,float(arrival))+service_ms`.\n ready=max(ready,float(arrival))+service_ms\n finish.append(ready)\n # Return `np.asarray(finish) - arrivals` to the caller.\n return np.asarray(finish)-arrivals" } ], "rubric": [ "Identify the trained base and adaptation data using recorded hashes", "Compare a stable objective and gradient with an independent reference", "Reload actual exported artifacts across declared batch and precision policies", "Keep calibration separate from evaluation and report activation clipping", "Verify corruption and malformed-request rejection", "Measure completion inside an explicit local request boundary", "Explain simulated queue growth without claiming a real server load test" ], "workspaceFile": "projects/deployment-audit/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/deployment-audit/starter/model.py projects/deployment-audit/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/deployment-audit/starter/model.py projects/deployment-audit/my_model.py" } ], "referenceCommand": "python3 projects/deployment-audit/tests/check.py --implementation solution --stage all", "setupNote": "Use the top folder of the extracted ZIP as the working directory. It contains requirements-cpu.txt and projects/deployment-audit. This project exercises an in-process CPU request boundary; real network and target-device measurements require separate runs. Copy the starter scaffold `projects/deployment-audit/starter/model.py` to `projects/deployment-audit/my_model.py`, implement the stage functions in `projects/deployment-audit/my_model.py`, and verify each stage with `python3 projects/deployment-audit/tests/check.py --implementation projects/deployment-audit/my_model.py --stage 1` (through `--stage 3`).", "assessmentId": "ship" }, { "schemaVersion": 1, "id": "workload-operations", "title": "Operate and recover a local JAX workload", "pathwayId": "ops", "status": "authored", "hardware": "CPU; optional target runtime requires separate validation", "source": "projects/workload-operations/README.md", "requiredLessonIds": [ "operations-01", "operations-02", "operations-03", "operations-04", "operations-05" ], "stages": [ { "id": "1", "title": "Real process lifecycle", "command": "python3 projects/workload-operations/tests/check.py --stage 1 --implementation projects/workload-operations/my_model.py", "evidence": "Success, runtime rejection and reaped timeout traces", "expected": "PASS stage 1", "diagnosis": "Check observed backend, last event, stderr and the actual child exit before retrying.", "codeGuide": "In `projects/workload-operations/my_model.py`, implement `digest(...)`, `atomic_json(...)` using `hashlib.sha256`, `json.dumps`, `encode`, `hexdigest`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def digest(value):\n \"\"\"SHA-256 of sorted compact JSON with nonfinite values rejected.\"\"\"\n # Key APIs to use: `hashlib.sha256`, `json.dumps`, `encode`, `hexdigest`\n # Step 1: Return `hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode()).hexdigest()` to the caller.\n raise NotImplementedError('Canonical content identity')\n\ndef atomic_json(path, value):\n \"\"\"Write/flush/fsync a sibling temporary file; replace the selected path.\"\"\"\n # Key APIs to use: `disk`, `Path`, `parent.mkdir`, `tempfile.mkstemp`, `os.fdopen`\n # Step 1: Read or serialize artifact data on disk (`path`).\n # Step 2: Execute the next step of the computation.\n # Step 3: Run `tempfile.mkstemp` to compute `(fd, temporary)`.\n raise NotImplementedError('Atomic local selection')", "solutionSnippet": "# Function `digest` reference implementation:\ndef digest(value):\n # Return `hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode()).hexdigest()` to the caller.\n return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode()).hexdigest()\n\n# Function `atomic_json` reference implementation:\ndef atomic_json(path, value):\n \"\"\"Replace one local JSON file only after its bytes have been flushed.\"\"\"\n # Compute `path` as `Path(path)`.\n path = Path(path)\n path.parent.mkdir(parents=True, exist_ok=True)\n # Run `tempfile.mkstemp` to compute `(fd, temporary)`.\n fd, temporary = tempfile.mkstemp(prefix=path.name+'.', suffix='.tmp', dir=path.parent)\n # Run the boundary check and catch the expected exception:\n try:\n # Enter `os.fdopen(fd, 'w')` context block:\n with os.fdopen(fd, 'w') as stream:\n json.dump(value, stream, sort_keys=True, allow_nan=False)\n stream.flush()\n os.fsync(stream.fileno())\n os.replace(temporary, path)\n finally:\n if os.path.exists(temporary):\n os.unlink(temporary)" }, { "id": "2", "title": "Workload observability", "command": "python3 projects/workload-operations/tests/check.py --stage 2 --implementation projects/workload-operations/my_model.py", "evidence": "Raw structured events and independently recomputed throughput boundaries", "expected": "PASS stage 2", "diagnosis": "Count examples once and synchronize completed updates before timing.", "codeGuide": "In `projects/workload-operations/my_model.py`, implement `default_config(...)`, `launch(...)` using `cfg.update`, `in`, `contract`, `or`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def default_config(**changes):\n \"\"\"Return validated defaults plus changes. See README for default values.\"\"\"\n # Key APIs to use: `cfg.update`, `in`, `contract`, `or`\n # Step 1: Evaluate `seed=3, learning_rate=0.04, momentum=0.8, batch_size=8, steps=8, checkpoint_every=2, backend='cpu', min_devices=1, resume=False, fail_after=None, stall_at=None, stall_seconds=10.0` and convert the result into Python scalar/collection `cfg`.\n # Step 2: Update state in place with the new values.\n # Step 3: Loop over `name` in `('steps', 'checkpoint_every', 'batch_size', 'min_devices')`:\n # Step 4: Inside block: Guard input contract (`not isinstance(cfg[name], int) or isinstance(cfg[name], bool) or cfg[name] < 1`) and fail fast if violated.\n # Step 5: Guard input contract (`cfg['batch_size'] > 64 or not 0 <= cfg['momentum'] < 1 or (not 0 < cfg['learning_rate'] < 1)`) and fail fast if violated.\n # Step 6: Return `cfg` to the caller.\n raise NotImplementedError('Validate the workload contract')\n\ndef launch(root, config=None, timeout=10.):\n \"\"\"Write worker/config, run a real child, capture events, and reap on timeout.\n\n Return status, returncode, wall_s, events, stderr, worker_hash, config and\n malformed_stdout\n atomically retain the same record in root/run.json.\n \"\"\"\n # Key APIs to use: `disk`, `Path`, `root.mkdir`, `default_config`, `contract`\n # Step 1: Read or serialize artifact data on disk (`root`).\n # Step 2: Execute the next step of the computation.\n # Step 3: Run `default_config` to compute `cfg`.\n # Step 4: Guard input contract (`timeout <= 0`) and fail fast if violated.\n # Step 5: Run `atomic_json` to perform the next check or state transition.\n # Step 6: Evaluate `worker` from the current inputs and state.\n raise NotImplementedError('Supervise the process, not an invented status')", "solutionSnippet": "# Function `default_config` reference implementation:\ndef default_config(**changes):\n # Evaluate `seed=3, learning_rate=0.04, momentum=0.8, batch_size=8, steps=8, checkpoint_every=2, backend='cpu', min_devices=1, resume=False, fail_after=None, stall_at=None, stall_seconds=10.0` and convert the result into Python scalar/collection `cfg`.\n cfg=dict(seed=3,learning_rate=.04,momentum=.8,batch_size=8,steps=8,\n checkpoint_every=2,backend='cpu',min_devices=1,resume=False,\n fail_after=None,stall_at=None,stall_seconds=10.)\n # Update state in place with the new values.\n cfg.update(changes)\n # Iterate over `name` to step through the computation:\n for name in ('steps','checkpoint_every','batch_size','min_devices'):\n # Guard input contract (`not isinstance(cfg[name], int) or isinstance(cfg[name], bool) or cfg[name] < 1`) and fail fast if violated.\n if not isinstance(cfg[name],int) or isinstance(cfg[name],bool) or cfg[name] < 1:\n raise ValueError(name+' must be a positive integer')\n # Guard input contract (`cfg['batch_size'] > 64 or not 0 <= cfg['momentum'] < 1 or (not 0 < cfg['learning_rate'] < 1)`) and fail fast if violated.\n if cfg['batch_size'] > 64 or not 0 <= cfg['momentum'] < 1 or not 0 < cfg['learning_rate'] < 1:\n raise ValueError('invalid training configuration')\n # Return `cfg` to the caller.\n return cfg\n\n# Function `launch` reference implementation:\ndef launch(root, config=None, timeout=10.):\n # Compute `root` as `Path(root)`.\n root=Path(root)\n root.mkdir(parents=True,exist_ok=True)\n # Run `default_config` to compute `cfg`.\n cfg=default_config(**(config or {}))\n # Guard input contract (`timeout <= 0`) and fail fast if violated.\n if timeout <= 0:\n raise ValueError('timeout must be positive')\n # Run `atomic_json` to perform the next check or state transition.\n atomic_json(root/'config.json',cfg)\n # Compute `worker` as `root/'worker.py'`.\n worker=root/'worker.py'\n worker.write_text(WORKER)\n # Configure environment variable before initializing the runtime.\n env=dict(os.environ,JAX_PLATFORMS=cfg['backend'],PYTHONUNBUFFERED='1')\n # Record execution timing or profiler trace in `started`.\n started=time.perf_counter()\n # Read or serialize artifact data on disk (`process`).\n process=subprocess.Popen([sys.executable,str(worker),str(root)],stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True,env=env)\n # Compute `timed_out` as `False`.\n timed_out=False\n try:\n stdout,stderr=process.communicate(timeout=timeout)\n except subprocess.TimeoutExpired:\n timed_out=True\n process.terminate()\n try:\n stdout,stderr=process.communicate(timeout=1.)\n except subprocess.TimeoutExpired:\n process.kill()\n stdout,stderr=process.communicate(timeout=1.)\n # Record execution timing or profiler trace in `wall`.\n wall=time.perf_counter()-started\n # Initialize list `events` for the stage values.\n events=[]\n malformed=[]\n # Loop over `line` in `stdout.splitlines()`:\n for line in stdout.splitlines():\n # Branch on condition `not line.strip()`:\n if not line.strip():\n continue\n try:\n event=json.loads(line)\n if not isinstance(event,dict) or 'event' not in event:\n raise ValueError('not an event')\n events.append(event)\n except (ValueError,TypeError):\n malformed.append(line)\n # Compute `status` as `'timed_out' if timed_out else 'completed' if process.returncode == 0 and not malfo`.\n status='timed_out' if timed_out else 'completed' if process.returncode == 0 and not malformed and events and events[-1]['event']=='completed' else 'failed'\n # Compute deterministic cryptographic digest `result` for provenance verification.\n result=dict(status=status,returncode=process.returncode,wall_s=wall,events=events,stderr=stderr,\n worker_hash=hashlib.sha256(WORKER.encode()).hexdigest(),config=cfg,malformed_stdout=malformed)\n # Run `atomic_json` to perform the next check or state transition.\n atomic_json(root/'run.json',result)\n # Return `result` to the caller.\n return result" }, { "id": "3", "title": "Complete-state recovery", "command": "python3 projects/workload-operations/tests/check.py --stage 3 --implementation projects/workload-operations/my_model.py", "evidence": "Interrupted/uninterrupted next-update and final-state equality", "expected": "PASS stage 3", "diagnosis": "Inspect momentum, key, step and provenance rather than only final loss.", "codeGuide": "In `projects/workload-operations/my_model.py`, implement `summarize(...)`, `publish(...)` using `sum`, `max`, `disk`, `Path`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def summarize(result):\n \"\"\"Return updates/examples/last_step/checkpoint_step/uncheckpointed_updates,\n update_s, update_examples_per_s, job_examples_per_s, update_duty_fraction\n and status. Derive every value from actual events and wall_s.\n \"\"\"\n # Key APIs to use: `sum`, `max`\n # Step 1: Evaluate `updates` from the current inputs and state.\n # Step 2: Evaluate `checkpoints` from the current inputs and state.\n # Step 3: Run `sum` to compute `count`.\n # Step 4: Run `sum` to compute `update_s`.\n # Step 5: Run `max` to compute `latest`.\n # Step 6: Run `max` to compute `checkpoint`.\n raise NotImplementedError('Work and timing boundaries')\n\ndef publish(store, artifact):\n \"\"\"Write artifacts/.json without changing active.json; return digest.\"\"\"\n # Key APIs to use: `disk`, `Path`, `digest`, `atomic_json`\n # Step 1: Read or serialize artifact data on disk (`store`).\n # Step 2: Run `digest` to compute `identifier`.\n # Step 3: Run `atomic_json` to perform the next check or state transition.\n # Step 4: Return `identifier` to the caller.\n raise NotImplementedError('Publish independently of activation')", "solutionSnippet": "# Function `summarize` reference implementation:\ndef summarize(result):\n # Initialize list `updates` for the stage values.\n updates=[e for e in result['events'] if e['event']=='progress']\n # Initialize list `checkpoints` for the stage values.\n checkpoints=[e['step'] for e in result['events'] if e['event']=='checkpoint']\n # Run `sum` to compute `count`.\n # Run `sum` to compute `update_s`.\n count=sum(e['examples'] for e in updates)\n update_s=sum(e['update_s'] for e in updates)\n # Run `max` to compute `latest`.\n latest=max((e['step'] for e in updates),default=0)\n # Run `max` to compute `checkpoint`.\n checkpoint=max(checkpoints,default=0)\n # Return `dict(status=result['status'], updates=len(updates), examples=count, last_step=latest, checkpoint_step=checkpoint, uncheckpointed_updates=max(0, latest - checkpoint), update_s=update_s, update_examples_per_s=count / update_s if update_s else 0.0, job_examples_per_s=count / result['wall_s'] if result['wall_s'] else 0.0, update_duty_fraction=update_s / result['wall_s'] if result['wall_s'] else 0.0)` to the caller.\n return dict(status=result['status'],updates=len(updates),examples=count,last_step=latest,checkpoint_step=checkpoint,\n uncheckpointed_updates=max(0,latest-checkpoint),update_s=update_s,\n update_examples_per_s=count/update_s if update_s else 0.,\n job_examples_per_s=count/result['wall_s'] if result['wall_s'] else 0.,\n update_duty_fraction=update_s/result['wall_s'] if result['wall_s'] else 0.)\n\n# Function `publish` reference implementation:\ndef publish(store, artifact):\n \"\"\"Content-address one JSON artifact. Publishing does not activate it.\"\"\"\n # Run `digest` to compute `identifier`.\n store=Path(store)\n identifier=digest(artifact)\n # Run `atomic_json` to perform the next check or state transition.\n atomic_json(store/'artifacts'/(identifier+'.json'),artifact)\n # Return `identifier` to the caller.\n return identifier" }, { "id": "4", "title": "Artifact selection and rollback", "command": "python3 projects/workload-operations/tests/check.py --stage 4 --implementation projects/workload-operations/my_model.py", "evidence": "Content IDs, failed gate and unchanged pointer, tested rollback", "expected": "PASS stage 4", "diagnosis": "Verify stored payload digests and prior compatibility before changing the pointer.", "codeGuide": "In `projects/workload-operations/my_model.py`, implement `read_artifact(...)`, `artifact_metrics(...)` using `contract`, `any`, `disk`, `json.loads`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 4 verifier.", "starterSnippet": "def read_artifact(store, identifier):\n \"\"\"Require a 64-character lowercase hex ID and verify payload digest.\"\"\"\n # Key APIs to use: `contract`, `any`, `disk`, `json.loads`, `Path`\n # Step 1: Guard input contract (`len(identifier) != 64 or any((c not in '0123456789abcdef' for c in identifier))`) and fail fast if violated.\n # Step 2: Read or serialize artifact data on disk (`artifact`).\n # Step 3: Guard input contract (`digest(artifact) != identifier`) and fail fast if violated.\n # Step 4: Return `artifact` to the caller.\n raise NotImplementedError('Read with integrity checks')\n\ndef artifact_metrics(artifact):\n \"\"\"Evaluate params on x=(-1.5,-.37,.22,1.5), y=2*x+1; MSE limit 1.\"\"\"\n # Key APIs to use: `artifact.get`, `get`, `contract`, `any`, `math.isfinite`\n # Step 1: Run `artifact.get` to compute `params`.\n # Step 2: Guard input contract (`len(params) != 2 or any((not math.isfinite(float(v)) for v in params))`) and fail fast if violated.\n # Step 3: Evaluate `inputs` from the current inputs and state.\n # Step 4: Run `sum` to compute `mse`.\n # Step 5: Return `dict(held_out_mse=mse, acceptance_limit=1.0, passed=mse <= 1.0)` to the caller.\n raise NotImplementedError('Compute the actual release quality gate')", "solutionSnippet": "# Function `read_artifact` reference implementation:\ndef read_artifact(store, identifier):\n # Guard input contract (`len(identifier) != 64 or any((c not in '0123456789abcdef' for c in identifier))`) and fail fast if violated.\n if len(identifier) != 64 or any(c not in '0123456789abcdef' for c in identifier):\n raise ValueError('invalid artifact identifier')\n # Read or serialize artifact data on disk (`artifact`).\n artifact=json.loads((Path(store)/'artifacts'/(identifier+'.json')).read_text())\n # Guard input contract (`digest(artifact) != identifier`) and fail fast if violated.\n if digest(artifact) != identifier:\n raise ValueError('artifact digest mismatch')\n # Return `artifact` to the caller.\n return artifact\n\n# Function `artifact_metrics` reference implementation:\ndef artifact_metrics(artifact):\n \"\"\"Independent held-out arithmetic for this fixed linear-model release contract.\"\"\"\n # Run `artifact.get` to compute `params`.\n params=artifact.get('state',{}).get('params',[])\n # Guard input contract (`len(params) != 2 or any((not math.isfinite(float(v)) for v in params))`) and fail fast if violated.\n if len(params)!=2 or any(not math.isfinite(float(v)) for v in params):\n raise ValueError('invalid model parameters')\n # Evaluate the compound expression for `inputs`.\n inputs=(-1.5,-.37,.22,1.5)\n # Run `sum` to compute `mse`.\n mse=sum((params[0]*x+params[1]-(2*x+1))**2 for x in inputs)/len(inputs)\n # Return `dict(held_out_mse=mse, acceptance_limit=1.0, passed=mse <= 1.0)` to the caller.\n return dict(held_out_mse=mse,acceptance_limit=1.,passed=mse<=1.)" }, { "id": "5", "title": "Capacity and cost", "command": "python3 projects/workload-operations/tests/check.py --stage 5 --implementation projects/workload-operations/my_model.py", "evidence": "Measured local service time plus separately labeled demand, reserve and price assumptions", "expected": "PASS stage 5", "diagnosis": "Carry seconds-to-hours conversion, idle reservation and failure overhead explicitly.", "codeGuide": "In `projects/workload-operations/my_model.py`, implement `activate(...)`, `rollback(...)`, `capacity(...)` using `read_artifact`, `contract`, `artifact.get`, `get`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 5 verifier.", "starterSnippet": "def activate(store, identifier):\n \"\"\"Verify digest, provenance, declared validation and actual held-out MSE, then select.\n active.json contains current and previous content IDs.\n \"\"\"\n # Key APIs to use: `read_artifact`, `contract`, `artifact.get`, `get`, `any`\n # Step 1: Run `read_artifact` to compute `artifact`.\n # Step 2: Guard input contract (`artifact.get('validation', {}).get('passed') is not True`) and fail fast if violated.\n # Step 3: Evaluate `required` from the current inputs and state.\n # Step 4: Guard input contract (`any((name not in artifact for name in required))`) and fail fast if violated.\n # Step 5: Guard input contract (`not artifact_metrics(artifact)['passed']`) and fail fast if violated.\n # Step 6: Evaluate `previous` from the current inputs and state.\n raise NotImplementedError('Gate before pointer mutation')\n\ndef rollback(store):\n \"\"\"Select previous through the same checked activation path.\"\"\"\n # Key APIs to use: `disk`, `json.loads`, `Path`, `read_text`, `contract`\n # Step 1: Read or serialize artifact data on disk (`pointer`).\n # Step 2: Guard input contract (`not pointer.get('previous')`) and fail fast if violated.\n # Step 3: Return `activate(store, pointer['previous'])` to the caller.\n raise NotImplementedError('Checked rollback')\n\ndef capacity(measured_job_s, arrivals_per_hour, workers, hourly_rate, reserve_fraction=.25):\n \"\"\"Return offered_worker_hours_per_hour, load_fraction,\n minimum_workers_with_reserve, hourly_budget, hypothetical_cost_per_job,\n within_reserve. Validate finite input and document assumptions.\n \"\"\"\n # Key APIs to use: `contract`, `any`, `math.isfinite`, `or`, `max`\n # Step 1: Evaluate `values` from the current inputs and state.\n # Step 2: Guard input contract (`any((not math.isfinite(x) for x in values)) or measured_job_s <= 0 or arrivals_per_hour < 0 or (hourly_rate < 0)`) and fail fast if violated.\n # Step 3: Guard input contract (`not isinstance(workers, int) or isinstance(workers, bool) or workers < 1 or (not 0 <= reserve_fraction < 1)`) and fail fast if violated.\n # Step 4: Evaluate `offered` from the current inputs and state.\n # Step 5: Evaluate `load` from the current inputs and state.\n # Step 6: Run `max` to compute `required`.\n raise NotImplementedError('Dimensional capacity arithmetic')", "solutionSnippet": "# Function `activate` reference implementation:\ndef activate(store, identifier):\n # Run `read_artifact` to compute `artifact`.\n artifact=read_artifact(store,identifier)\n # Guard input contract (`artifact.get('validation', {}).get('passed') is not True`) and fail fast if violated.\n if artifact.get('validation',{}).get('passed') is not True:\n raise ValueError('artifact has no passing validation evidence')\n # Initialize list `required` for the stage values.\n required=['worker_hash','config_hash','data_hash','state']\n # Guard input contract (`any((name not in artifact for name in required))`) and fail fast if violated.\n if any(name not in artifact for name in required):\n raise ValueError('artifact provenance incomplete')\n # Guard input contract (`not artifact_metrics(artifact)['passed']`) and fail fast if violated.\n if not artifact_metrics(artifact)['passed']:\n raise ValueError('held-out model validation failed')\n # Read or serialize artifact data on disk (`pointer`).\n previous=None\n pointer=Path(store)/'active.json'\n # Branch on condition `pointer.exists()`:\n if pointer.exists():previous=json.loads(pointer.read_text())['current']\n # Run `atomic_json` to perform the next check or state transition.\n atomic_json(pointer,dict(current=identifier,previous=previous))\n # Return `identifier` to the caller.\n return identifier\n\n# Function `rollback` reference implementation:\ndef rollback(store):\n # Read or serialize artifact data on disk (`pointer`).\n pointer=json.loads((Path(store)/'active.json').read_text())\n # Guard input contract (`not pointer.get('previous')`) and fail fast if violated.\n if not pointer.get('previous'):raise ValueError('no previous version')\n # Return `activate(store, pointer['previous'])` to the caller.\n return activate(store,pointer['previous'])\n\n# Function `capacity` reference implementation:\ndef capacity(measured_job_s, arrivals_per_hour, workers, hourly_rate, reserve_fraction=.25):\n \"\"\"Illustrative one-job-per-worker plan; inputs are not cloud price discovery.\"\"\"\n # Initialize list `values` for the stage values.\n values=[measured_job_s,arrivals_per_hour,hourly_rate,reserve_fraction]\n # Guard input contract (`any((not math.isfinite(x) for x in values)) or measured_job_s <= 0 or arrivals_per_hour < 0 or (hourly_rate < 0)`) and fail fast if violated.\n if any(not math.isfinite(x) for x in values) or measured_job_s <= 0 or arrivals_per_hour < 0 or hourly_rate < 0:\n raise ValueError('invalid finite capacity inputs')\n # Guard input contract (`not isinstance(workers, int) or isinstance(workers, bool) or workers < 1 or (not 0 <= reserve_fraction < 1)`) and fail fast if violated.\n if not isinstance(workers,int) or isinstance(workers,bool) or workers < 1 or not 0 <= reserve_fraction < 1:\n raise ValueError('invalid workers or reserve')\n # Compute `offered` as `arrivals_per_hour*measured_job_s/3600`.\n offered=arrivals_per_hour*measured_job_s/3600\n # Compute `load` as `offered/workers`.\n load=offered/workers\n # Run `max` to compute `required`.\n required=max(1,math.ceil(offered/(1-reserve_fraction)))\n # Return `dict(offered_worker_hours_per_hour=offered, load_fraction=load, minimum_workers_with_reserve=required, hourly_budget=workers * hourly_rate, hypothetical_cost_per_job=measured_job_s * hourly_rate / 3600, within_reserve=load <= 1 - reserve_fraction)` to the caller.\n return dict(offered_worker_hours_per_hour=offered,load_fraction=load,\n minimum_workers_with_reserve=required,hourly_budget=workers*hourly_rate,\n hypothetical_cost_per_job=measured_job_s*hourly_rate/3600,\n within_reserve=load <= 1-reserve_fraction)" } ], "rubric": [ "Observe actual worker exit and reap timed-out processes", "Derive useful metrics from synchronized work with explicit units", "Rehearse failure and verify complete-state next-update recovery", "Reject corrupted or incompatible checkpoint provenance", "Publish, gate, atomically select and roll back exact content IDs", "Separate measured workload duration from hypothetical demand and prices", "Keep a runbook grounded in the actual failure trace and target limitations" ], "workspaceFile": "projects/workload-operations/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/workload-operations/starter/model.py projects/workload-operations/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/workload-operations/starter/model.py projects/workload-operations/my_model.py" } ], "referenceCommand": "python3 projects/workload-operations/tests/check.py --stage 5 --implementation solution", "setupNote": "Work from the extracted top folder containing requirements-cpu.txt and projects/workload-operations. The checks use bounded local subprocesses and temporary directories; no cloud allocation occurs. Copy the starter scaffold `projects/workload-operations/starter/model.py` to `projects/workload-operations/my_model.py`, implement the stage functions in `projects/workload-operations/my_model.py`, and verify each stage with `python3 projects/workload-operations/tests/check.py --implementation projects/workload-operations/my_model.py --stage 1` (through `--stage 5`).", "assessmentId": "ops" }, { "schemaVersion": 1, "id": "audio-harness", "title": "Build an audio waveform-to-inference harness", "pathwayId": "ship", "status": "authored", "hardware": "CPU; target-edge validation is a separate extension", "source": "projects/audio-harness/README.md", "requiredLessonIds": [ "arrays-02", "state-01", "state-02", "optimization-02", "networks-03", "recovery-03", "recovery-04", "deployment-03", "deployment-05", "deployment-06" ], "stages": [ { "id": "1", "title": "Waveform and spectral contract", "command": "python3 projects/audio-harness/tests/check.py --implementation projects/audio-harness/my_model.py --stage 1", "evidence": "Recording/split provenance, independent STFT, silence and tone-bin checks, actual WAV loading", "expected": "PASS stage 1: waveform contracts, independent STFT, silence, frequency bins, actual WAV provenance and split isolation", "diagnosis": "Check sample rate, channel axis, crop length, Hann convention and power normalization before changing the classifier.", "codeGuide": "In `projects/audio-harness/my_model.py`, implement `features(...)`, `load_wav_manifest(...)` using `features_core`, `jnp.asarray`, `validate_waveforms`, `disk`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def features(waveforms,sample_rate=8000,channels=1,version=\"audio-stft-v1\"):\n \"\"\"Stage 1: validate mono PCM (batch,1024), 8 kHz, finite amplitudes [-1,1].\n Symmetric Hann 128, hop 64, 15 frames, rFFT 65 bins, power divided by window\n squared sum. Return mean(log1p(power), frame axis), shape (batch,65).\n \"\"\"\n # Key APIs to use: `features_core`, `jnp.asarray`, `validate_waveforms`\n # Step 1: Return `features_core(jnp.asarray(validate_waveforms(waveforms, sample_rate, channels, version)))` to the caller.\n raise NotImplementedError(\"Implement and independently verify STFT features\")\n\ndef load_wav_manifest(manifest_path,split):\n \"\"\"Stage 1: PCM16 mono 8 kHz WAV rows with id/file/label/split/group/license/source/sha256.\n Verify bytes, reject group leakage, select 1280 samples at declared offset_samples.\n \"\"\"\n # Key APIs to use: `disk`, `Path`, `json.loads`, `manifest_path.read_text`, `in`\n # Step 1: Read or serialize artifact data on disk (`manifest_path`).\n # Step 2: Read or serialize artifact data on disk (`rows`).\n # Step 3: Evaluate `groups` from the current inputs and state.\n # Step 4: Run `set` to compute `ids`.\n # Step 5: Loop over `row` in `rows`:\n # Step 6: Inside block: Loop over `field` in `('id', 'file', 'label', 'split', 'group', 'license', 'source', 'sha256')`:\n raise NotImplementedError(\"Preserve recording rights, identity and split provenance\")", "solutionSnippet": "# Function `features` reference implementation:\ndef features(waveforms,sample_rate=8000,channels=1,version=\"audio-stft-v1\"):\n # Return `features_core(jnp.asarray(validate_waveforms(waveforms, sample_rate, channels, version)))` to the caller.\n return features_core(jnp.asarray(validate_waveforms(waveforms,sample_rate,channels,version)))\n\n# Function `load_wav_manifest` reference implementation:\ndef load_wav_manifest(manifest_path,split):\n \"\"\"Ingest user-provided licensed PCM16/mono/8kHz files; no implicit resampling.\"\"\"\n # Read or serialize artifact data on disk (`rows`).\n manifest_path=Path(manifest_path)\n rows=json.loads(manifest_path.read_text())\n # Run `set` to compute `ids`.\n groups={}\n ids=set()\n # Loop over `row` in `rows`:\n for row in rows:\n # Loop over `field` in `('id', 'file', 'label', 'split', 'group', 'license', 'source', 'sha256')`:\n for field in (\"id\",\"file\",\"label\",\"split\",\"group\",\"license\",\"source\",\"sha256\"):\n # Guard input contract (`not row.get(field) and row.get(field) != 0`) and fail fast if violated.\n if not row.get(field) and row.get(field)!=0:raise ValueError(\"missing provenance field: \"+field)\n # Guard input contract (`row['id'] in ids`) and fail fast if violated.\n if row[\"id\"] in ids:raise ValueError(\"duplicate recording identity\")\n # Run `ids.add` to perform the next check or state transition.\n # Run `ids.add` to perform the next check or state transition.\n ids.add(row[\"id\"])\n groups.setdefault(row[\"group\"],set()).add(row[\"split\"])\n # Guard input contract (`any((len(splits) > 1 for splits in groups.values()))`) and fail fast if violated.\n if any(len(splits)>1 for splits in groups.values()):raise ValueError(\"recording group leaks across splits\")\n # Initialize list `output` for the stage values.\n output=[]\n labels=[]\n selected=[]\n # Loop over `row` in `rows`:\n for row in rows:\n # Branch on condition `row['split'] != split`:\n if row[\"split\"]!=split:continue\n # Compute `path` as `manifest_path.parent/row[\"file\"]`.\n path=manifest_path.parent/row[\"file\"]\n # Guard input contract (`hashlib.sha256(path.read_bytes()).hexdigest() != row['sha256']`) and fail fast if violated.\n if hashlib.sha256(path.read_bytes()).hexdigest()!=row[\"sha256\"]:raise ValueError(\"WAV checksum mismatch\")\n # Enter managed runtime/context scope for this block:\n with wave.open(str(path),\"rb\") as wav:\n # Guard input contract (`wav.getframerate() != 8000 or wav.getnchannels() != 1 or wav.getsampwidth() != 2`) and fail fast if violated.\n if wav.getframerate()!=8000 or wav.getnchannels()!=1 or wav.getsampwidth()!=2:\n raise ValueError(\"require PCM16 mono 8kHz; convert explicitly and preserve source provenance\")\n # Cast or evaluate `samples` in explicit floating-point precision.\n samples=np.frombuffer(wav.readframes(wav.getnframes()),dtype=\" 1) or (np.asarray(data['labels']).dtype.kind not in 'iu') or np.any((data['labels'] < 0) | (data['labels'] > 2))`) and fail fast if violated.\n # Step 3: Guard input contract (`np.asarray(data['waveforms']).shape != (len(data['labels']), 1280)`) and fail fast if violated.\n # Step 4: Guard input contract (`batch_size < 1 or rate <= 0 or (not 0 <= momentum < 1)`) and fail fast if violated.\n # Step 5: Run `features` to compute `train_features`.\n # Step 6: Reduce across the target axis to summarize `mean`.\n raise NotImplementedError(\"Create an explicit resumable state\")\n\ndef step(state,data):\n \"\"\"Stage 2: one crop/gain/noise-augmented minibatch and momentum update.\n Return new state plus IDs/crop_offsets/gains/feature_sha256/loss/completed_step.\n \"\"\"\n # Key APIs to use: `contract`, `dataset_hash`, `random.split`, `random.permutation`, `np.asarray`\n # Step 1: Guard input contract (`state['dataset_sha256'] != dataset_hash(data)`) and fail fast if violated.\n # Step 2: Evaluate `state` and convert the result into Python scalar/collection `state`.\n # Step 3: Branch on condition `state['cursor'] == len(data['labels'])`:\n # Step 4: Evaluate `cursor` from the current inputs and state.\n # Step 5: Convert `indices` to a host NumPy array for inspection or verification.\n # Step 6: Split the PRNG key deterministically into independent subkeys (`(state['key'], crop_key, gain_key, noise_key)`).\n raise NotImplementedError(\"Keep sampling, augmentation and optimizer state explicit\")\n\ndef save_checkpoint(path,state):\n \"\"\"Stage 2: NPZ numeric arrays + JSON contract/hash; no executable pickle.\"\"\"\n # Key APIs to use: `disk`, `Path`, `path.mkdir`, `np.savez`, `np.asarray`\n # Step 1: Read or serialize artifact data on disk (`path`).\n # Step 2: Execute the next step of the computation.\n # Step 3: Convert `` to a host NumPy array for inspection or verification.\n # Step 4: Compute deterministic cryptographic digest `manifest` for provenance verification.\n # Step 5: Compute deterministic cryptographic digest `` for provenance verification.\n # Execute `raise NotImplementedError(\"Save the complete committed training boundary\")`.\n raise NotImplementedError(\"Save the complete committed training boundary\")\n\ndef load_checkpoint(path,data,config):\n \"\"\"Stage 2: reject changed data, preprocessing, runtime, optimizer configuration or bytes.\"\"\"\n # Key APIs to use: `disk`, `Path`, `json.loads`, `read_text`, `contract`\n # Step 1: Read or serialize artifact data on disk (`path`).\n # Step 2: Read or serialize artifact data on disk (`manifest`).\n # Step 3: Guard input contract (`manifest['dataset_sha256'] != dataset_hash(data) or manifest['config'] != config`) and fail fast if violated.\n # Step 4: Guard input contract (`manifest['preprocessing'] != PREPROCESS or manifest['jax'] != jax.__version__`) and fail fast if violated.\n # Step 5: Guard input contract (`hashlib.sha256((path / 'state.npz').read_bytes()).hexdigest() != manifest['state_sha256']`) and fail fast if violated.\n # Step 6: Enter managed runtime/context scope for this block:\n raise NotImplementedError(\"Restore only a matching training experiment\")\n\ndef evaluate(state,data,batch_size=13):\n \"\"\"Stage 2: fixed center windows; return count/loss_sum/loss/correct/accuracy/confusion.\"\"\"\n # Key APIs to use: `np.zeros`, `fixed_windows`, `np.asarray`, `scores`, `logits.max`\n # Step 1: Evaluate `count` from the current inputs and state.\n # Step 2: Evaluate `loss_sum` from the current inputs and state.\n # Step 3: Evaluate `correct` from the current inputs and state.\n # Step 4: Allocate initialized array `confusion` with the specified shape and dtype.\n # Step 5: Run `fixed_windows` to compute `windows`.\n # Step 6: Loop over `start` in `range(0, len(windows), batch_size)`:\n raise NotImplementedError(\"Aggregate counts and sums without mutating training state\")", "solutionSnippet": "# Function `objective` reference implementation:\ndef objective(W,b,normalized,labels):\n # Guard input contract (`normalized.ndim != 2 or labels.shape != (normalized.shape[0],) or (not jnp.issubdtype(labels.dtype, jnp.integer))`) and fail fast if violated.\n if normalized.ndim!=2 or labels.shape!=(normalized.shape[0],) or not jnp.issubdtype(labels.dtype,jnp.integer):\n raise ValueError(\"one integer class label per feature row required\")\n # Perform matrix contraction / projection to compute `scores`.\n scores=normalized@W+b\n # Return `-jnp.mean(jax.nn.log_softmax(scores)[jnp.arange(len(labels)), labels])` to the caller.\n return -jnp.mean(jax.nn.log_softmax(scores)[jnp.arange(len(labels)),labels])\n\n# Function `create_state` reference implementation:\ndef create_state(data,seed=0,batch_size=12,rate=.04,momentum=.85):\n # Guard input contract (`data['sample_rate'] != 8000 or len(data['ids']) != len(data['labels'])`) and fail fast if violated.\n if data[\"sample_rate\"]!=8000 or len(data[\"ids\"])!=len(data[\"labels\"]):\n raise ValueError(\"dataset contract mismatch\")\n # Guard input contract (`len(data['groups']) != len(data['ids']) or len(set(data['ids'])) != len(data['ids']) or (not np.isfinite(data['waveforms']).all()) or (np.max(np.abs(data['waveforms'])) > 1) or (np.asarray(data['labels']).dtype.kind not in 'iu') or np.any((data['labels'] < 0) | (data['labels'] > 2))`) and fail fast if violated.\n if (len(data[\"groups\"])!=len(data[\"ids\"]) or len(set(data[\"ids\"]))!=len(data[\"ids\"])\n or not np.isfinite(data[\"waveforms\"]).all() or np.max(np.abs(data[\"waveforms\"]))>1\n or np.asarray(data[\"labels\"]).dtype.kind not in \"iu\" or np.any((data[\"labels\"]<0)|(data[\"labels\"]>2))):\n raise ValueError(\"invalid recording identities, PCM values or class labels\")\n # Guard input contract (`np.asarray(data['waveforms']).shape != (len(data['labels']), 1280)`) and fail fast if violated.\n if np.asarray(data[\"waveforms\"]).shape!=(len(data[\"labels\"]),1280):\n raise ValueError(\"training recording shape must be (recordings,1280)\")\n # Guard input contract (`batch_size < 1 or rate <= 0 or (not 0 <= momentum < 1)`) and fail fast if violated.\n if batch_size<1 or rate<=0 or not 0<=momentum<1:\n raise ValueError(\"invalid optimizer or batch configuration\")\n # Run `features` to compute `train_features`.\n train_features=features(fixed_windows(data))\n # Reduce across the target axis to summarize `mean`.\n mean=train_features.mean(axis=0)\n # Reduce across the target axis to summarize `std`.\n std=jnp.maximum(train_features.std(axis=0),.05)\n # Initialize explicit deterministic PRNG key `(key, init_key, order_key)`.\n key,init_key,order_key=jax.random.split(jax.random.PRNGKey(seed),3)\n # Draw pseudorandom samples for `W` using the explicit RNG state.\n W=.01*jax.random.normal(init_key,(65,3))\n # Return `{'W': W, 'b': jnp.zeros(3), 'vW': jnp.zeros_like(W), 'vb': jnp.zeros(3), 'mean': mean, 'std': std, 'key': key, 'order': jax.random.permutation(order_key, len(data['labels'])), 'step': 0, 'cursor': 0, 'epoch': 0, 'config': {'seed': seed, 'batch_size': batch_size, 'rate': rate, 'momentum': momentum}, 'dataset_sha256': dataset_hash(data)}` to the caller.\n return {\"W\":W,\"b\":jnp.zeros(3),\"vW\":jnp.zeros_like(W),\"vb\":jnp.zeros(3),\n \"mean\":mean,\"std\":std,\"key\":key,\"order\":jax.random.permutation(order_key,len(data[\"labels\"])),\n \"step\":0,\"cursor\":0,\"epoch\":0,\n \"config\":{\"seed\":seed,\"batch_size\":batch_size,\"rate\":rate,\"momentum\":momentum},\n \"dataset_sha256\":dataset_hash(data)}\n\n# Function `step` reference implementation:\ndef step(state,data):\n # Guard input contract (`state['dataset_sha256'] != dataset_hash(data)`) and fail fast if violated.\n if state[\"dataset_sha256\"]!=dataset_hash(data):\n raise ValueError(\"training dataset identity changed\")\n # Evaluate `state` and convert the result into Python scalar/collection `state`.\n state=dict(state)\n # Branch on condition `state['cursor'] == len(data['labels'])`:\n if state[\"cursor\"]==len(data[\"labels\"]):\n state[\"key\"],order_key=jax.random.split(state[\"key\"])\n state[\"order\"]=jax.random.permutation(order_key,len(data[\"labels\"]))\n state[\"cursor\"]=0\n state[\"epoch\"]+=1\n # Compute `cursor` as `state[\"cursor\"]`.\n cursor=state[\"cursor\"]\n # Convert `indices` to a host NumPy array for inspection or verification.\n indices=np.asarray(state[\"order\"][cursor:cursor+state[\"config\"][\"batch_size\"]])\n # Split the PRNG key deterministically into independent subkeys (`(state['key'], crop_key, gain_key, noise_key)`).\n state[\"key\"],crop_key,gain_key,noise_key=jax.random.split(state[\"key\"],4)\n # Draw pseudorandom samples for `offsets` using the explicit RNG state.\n offsets=jax.random.randint(crop_key,(len(indices),),0,257)\n # Draw pseudorandom samples for `gains` using the explicit RNG state.\n gains=jax.random.uniform(gain_key,(len(indices),1),minval=.8,maxval=1.15)\n # Create device-backed JAX array `clips`.\n clips=jnp.asarray(data[\"waveforms\"][indices])\n # Create evenly spaced index values in `windows`.\n windows=jnp.take_along_axis(clips,offsets[:,None]+jnp.arange(1024)[None,:],axis=1)\n # Draw pseudorandom samples for `augmented` using the explicit RNG state.\n augmented=windows*gains+.003*jax.random.normal(noise_key,windows.shape)\n # Run `features_core` to compute `feat`.\n feat=features_core(augmented)\n # Evaluate the compound expression for `normalized`.\n normalized=(feat-state[\"mean\"])/state[\"std\"]\n # Create device-backed JAX array `(state['W'], state['b'], state['vW'], state['vb'], loss)`.\n state[\"W\"],state[\"b\"],state[\"vW\"],state[\"vb\"],loss=update_arrays(\n state[\"W\"],state[\"b\"],state[\"vW\"],state[\"vb\"],normalized,\n jnp.asarray(data[\"labels\"][indices]),state[\"config\"][\"rate\"],state[\"config\"][\"momentum\"])\n # Accumulate the next contribution into `state['cursor']`.\n # Accumulate the next contribution into `state['step']`.\n state[\"cursor\"]+=len(indices)\n state[\"step\"]+=1\n # Convert `trace` to a host NumPy array for inspection or verification.\n trace={\"ids\":[data[\"ids\"][int(i)] for i in indices],\"crop_offsets\":np.asarray(offsets).tolist(),\n \"gains\":np.asarray(gains).ravel().tolist(),\"feature_sha256\":digest_arrays(np.asarray(feat)),\n \"loss\":float(loss),\"completed_step\":state[\"step\"]}\n # Return `(state, trace)` to the caller.\n return state,trace\n\n# Function `save_checkpoint` reference implementation:\ndef save_checkpoint(path,state):\n # Compute `path` as `Path(path)`.\n path=Path(path)\n path.mkdir(parents=True,exist_ok=True)\n # Convert `` to a host NumPy array for inspection or verification.\n np.savez(path/\"state.npz\",**{k:np.asarray(state[k]) for k in ARRAY_FIELDS})\n # Compute deterministic cryptographic digest `manifest` for provenance verification.\n manifest={k:state[k] for k in (\"step\",\"cursor\",\"epoch\",\"config\",\"dataset_sha256\")}\n # Compute deterministic cryptographic digest `` for provenance verification.\n manifest.update(preprocessing=PREPROCESS,jax=jax.__version__,\n state_sha256=hashlib.sha256((path/\"state.npz\").read_bytes()).hexdigest())\n # Compute `(path/\"checkpoint.json\").write_text(json.dumps(manifest,indent` as `2))`.\n (path/\"checkpoint.json\").write_text(json.dumps(manifest,indent=2))\n\n# Function `load_checkpoint` reference implementation:\ndef load_checkpoint(path,data,config):\n # Read or serialize artifact data on disk (`manifest`).\n path=Path(path)\n manifest=json.loads((path/\"checkpoint.json\").read_text())\n # Guard input contract (`manifest['dataset_sha256'] != dataset_hash(data) or manifest['config'] != config`) and fail fast if violated.\n if manifest[\"dataset_sha256\"]!=dataset_hash(data) or manifest[\"config\"]!=config:\n raise ValueError(\"dataset or training configuration mismatch\")\n # Guard input contract (`manifest['preprocessing'] != PREPROCESS or manifest['jax'] != jax.__version__`) and fail fast if violated.\n if manifest[\"preprocessing\"]!=PREPROCESS or manifest[\"jax\"]!=jax.__version__:\n raise ValueError(\"preprocessing or runtime mismatch\")\n # Guard input contract (`hashlib.sha256((path / 'state.npz').read_bytes()).hexdigest() != manifest['state_sha256']`) and fail fast if violated.\n if hashlib.sha256((path/\"state.npz\").read_bytes()).hexdigest()!=manifest[\"state_sha256\"]:\n raise ValueError(\"checkpoint hash mismatch\")\n # Enter managed runtime/context scope for this block:\n with np.load(path/\"state.npz\",allow_pickle=False) as data_arrays:\n # Create device-backed JAX array `state`.\n state={k:jnp.asarray(data_arrays[k]) for k in ARRAY_FIELDS}\n # Compute deterministic cryptographic digest `` for provenance verification.\n state.update({k:manifest[k] for k in (\"step\",\"cursor\",\"epoch\",\"config\",\"dataset_sha256\")})\n # Return `state` to the caller.\n return state\n\n# Function `evaluate` reference implementation:\ndef evaluate(state,data,batch_size=13):\n # Allocate initialized array `confusion` with the specified shape and dtype.\n count=0\n loss_sum=0.\n correct=0\n confusion=np.zeros((3,3),dtype=np.int64)\n # Run `fixed_windows` to compute `windows`.\n windows=fixed_windows(data)\n # Loop over `start` in `range(0, len(windows), batch_size)`:\n for start in range(0,len(windows),batch_size):\n # Compute `y` as `data[\"labels\"][start:start+batch_size]`.\n y=data[\"labels\"][start:start+batch_size]\n # Convert `logits` to a host NumPy array for inspection or verification.\n logits=np.asarray(scores(state,windows[start:start+batch_size]))\n # Reduce across the target axis to summarize `shifted`.\n shifted=logits-logits.max(axis=1,keepdims=True)\n # Reduce across the target axis to summarize `logp`.\n logp=shifted-np.log(np.exp(shifted).sum(axis=1,keepdims=True))\n # Run `logits.argmax` to compute `predicted`.\n predicted=logits.argmax(axis=1)\n # Accumulate the next contribution into `loss_sum`.\n # Accumulate the next contribution into `count`.\n loss_sum+=float(-logp[np.arange(len(y)),y].sum())\n count+=len(y)\n # Accumulate the next contribution into `correct`.\n correct+=int((predicted==y).sum())\n # Run `np.add.at` to perform the next check or state transition.\n np.add.at(confusion,(y,predicted),1)\n # Return `{'count': count, 'loss_sum': loss_sum, 'loss': loss_sum / count, 'correct': correct, 'accuracy': correct / count, 'confusion': confusion.tolist()}` to the caller.\n return {\"count\":count,\"loss_sum\":loss_sum,\"loss\":loss_sum/count,\n \"correct\":correct,\"accuracy\":correct/count,\"confusion\":confusion.tolist()}" }, { "id": "3", "title": "Precision and complete exported artifacts", "command": "python3 projects/audio-harness/tests/check.py --implementation projects/audio-harness/my_model.py --stage 3", "evidence": "Training-only calibration, integer arithmetic reference, six actual exports, fresh-process reload and failed-release rollback", "expected": "PASS stage 3: training-only calibration, integer oracle, six complete exports, fresh-process reload and corruption rejection", "diagnosis": "Inspect activation ranges, accumulator type and restored preprocessing; reject a mismatched or corrupt release.", "codeGuide": "In `projects/audio-harness/my_model.py`, implement `calibrate(...)`, `policy_scores(...)`, `export_release(...)`, `load_release(...)` using `contract`, `dataset_hash`, `features`, `fixed_windows`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def calibrate(state,training_data):\n \"\"\"Stage 3: train-only normalized-feature scale, per-class signed INT8 weights.\n Return qW, weight_scales, activation_scale, calibration_sha256.\n \"\"\"\n # Key APIs to use: `contract`, `dataset_hash`, `features`, `fixed_windows`, `np.asarray`\n # Step 1: Guard input contract (`dataset_hash(training_data) != state['dataset_sha256']`) and fail fast if violated.\n # Step 2: Run `features` to compute `feature`.\n # Step 3: Convert `normalized` to a host NumPy array for inspection or verification.\n # Step 4: Reduce across the target axis to summarize `activation_scale`.\n # Step 5: Convert `W` to a host NumPy array for inspection or verification.\n # Step 6: Reduce across the target axis to summarize `maximum`.\n raise NotImplementedError(\"Keep calibration separate from final evaluation\")\n\ndef policy_scores(state,waveforms,calibration,policy=\"fp32\"):\n \"\"\"Stage 3: raw waveform -> STFT -> fixed normalizer -> FP32/W8A32/W8A8 logits.\n W8A8 uses INT32 dot accumulation, then FP32 rescaling, bias and output.\n \"\"\"\n # Key APIs to use: `features_core`, `jnp.asarray`, `q.astype`, `contract`, `jnp.clip`\n # Step 1: Evaluate `x` from the current inputs and state.\n # Step 2: Branch on condition `policy == 'fp32'`:\n # Step 3: Create device-backed JAX array `q`.\n # Step 4: Create device-backed JAX array `s`.\n # Step 5: Branch on condition `policy == 'w8a32'`:\n # Step 6: Guard input contract (`policy != 'w8a8'`) and fail fast if violated.\n raise NotImplementedError(\"Specify every precision boundary\")\n\ndef export_release(path,state,calibration):\n \"\"\"Stage 3: six real JAX serialized exports: policies fp32/w8a32/w8a8, batches 1/4.\n Export must include waveform preprocessing and fitted normalization.\n \"\"\"\n # Key APIs to use: `disk`, `Path`, `path.mkdir`, `contract`, `in`\n # Step 1: Read or serialize artifact data on disk (`path`).\n # Step 2: Execute the next step of the computation.\n # Step 3: Guard input contract (`calibration['calibration_sha256'] != state['dataset_sha256']`) and fail fast if violated.\n # Step 4: Evaluate `artifacts` from the current inputs and state.\n # Step 5: Loop over `policy` in `('fp32', 'w8a32', 'w8a8')`:\n # Step 6: Inside block: Loop over `batch` in `BATCHES`:\n raise NotImplementedError(\"Package the connected computation and metadata\")\n\ndef load_release(path):\n \"\"\"Stage 3: validate release contract/hashes; return manifest and restored exports.\"\"\"\n # Key APIs to use: `disk`, `Path`, `json.loads`, `read_text`, `contract`\n # Step 1: Read or serialize artifact data on disk (`path`).\n # Step 2: Read or serialize artifact data on disk (`manifest`).\n # Step 3: Guard input contract (`manifest['preprocessing'] != PREPROCESS or manifest['jax'] != jax.__version__`) and fail fast if violated.\n # Step 4: Evaluate `artifacts` from the current inputs and state.\n # Step 5: Loop over `(key, item)` in `manifest['artifacts'].items()`:\n # Step 6: Inside block: Evaluate `data` from the current inputs and state.\n raise NotImplementedError(\"Load actual bytes in a fresh process\")", "solutionSnippet": "# Function `calibrate` reference implementation:\ndef calibrate(state,training_data):\n # Guard input contract (`dataset_hash(training_data) != state['dataset_sha256']`) and fail fast if violated.\n if dataset_hash(training_data)!=state[\"dataset_sha256\"]:\n raise ValueError(\"calibration must use declared training data, not evaluation data\")\n # Run `features` to compute `feature`.\n feature=features(fixed_windows(training_data))\n # Convert `normalized` to a host NumPy array for inspection or verification.\n normalized=np.asarray((feature-state[\"mean\"])/state[\"std\"])\n # Reduce across the target axis to summarize `activation_scale`.\n activation_scale=max(float(np.max(np.abs(normalized)))/127,1e-8)\n # Convert `W` to a host NumPy array for inspection or verification.\n # Reduce across the target axis to summarize `maximum`.\n W=np.asarray(state[\"W\"])\n maximum=np.max(np.abs(W),axis=0,keepdims=True)\n # Cast or evaluate `weight_scales` in explicit floating-point precision.\n weight_scales=np.where(maximum==0,1.,maximum/127).astype(np.float32)\n # Combine or mask array elements to form `q`.\n q=np.clip(np.rint(W/weight_scales),-127,127).astype(np.int8)\n # Return `{'qW': q, 'weight_scales': weight_scales, 'activation_scale': activation_scale, 'calibration_sha256': dataset_hash(training_data)}` to the caller.\n return {\"qW\":q,\"weight_scales\":weight_scales,\"activation_scale\":activation_scale,\n \"calibration_sha256\":dataset_hash(training_data)}\n\n# Function `policy_scores` reference implementation:\ndef policy_scores(state,waveforms,calibration,policy=\"fp32\"):\n # Evaluate the compound expression for `x`.\n x=(features_core(waveforms)-state[\"mean\"])/state[\"std\"]\n # Branch on condition `policy == 'fp32'`:\n if policy==\"fp32\":return x@state[\"W\"]+state[\"b\"]\n # Create device-backed JAX array `q`.\n # Create device-backed JAX array `s`.\n q=jnp.asarray(calibration[\"qW\"])\n s=jnp.asarray(calibration[\"weight_scales\"])\n # Branch on condition `policy == 'w8a32'`:\n if policy==\"w8a32\":return x@(q.astype(jnp.float32)*s)+state[\"b\"]\n # Guard input contract (`policy != 'w8a8'`) and fail fast if violated.\n if policy!=\"w8a8\":raise ValueError(\"unknown precision policy\")\n # Compute `a` as `calibration[\"activation_scale\"]`.\n a=calibration[\"activation_scale\"]\n # Combine or mask array elements to form `qa`.\n qa=jnp.clip(jnp.rint(x/a),-127,127).astype(jnp.int8)\n # Perform matrix contraction / projection to compute `accum`.\n accum=qa.astype(jnp.int32)@q.astype(jnp.int32)\n # Return `accum.astype(jnp.float32) * (a * s) + state['b']` to the caller.\n return accum.astype(jnp.float32)*(a*s)+state[\"b\"]\n\n# Function `export_release` reference implementation:\ndef export_release(path,state,calibration):\n # Compute `path` as `Path(path)`.\n path=Path(path)\n path.mkdir(parents=True,exist_ok=True)\n # Guard input contract (`calibration['calibration_sha256'] != state['dataset_sha256']`) and fail fast if violated.\n if calibration[\"calibration_sha256\"]!=state[\"dataset_sha256\"]:\n raise ValueError(\"calibration provenance mismatch\")\n # Construct dictionary `artifacts` with the structured fields for this stage.\n artifacts={}\n # Loop over `policy` in `('fp32', 'w8a32', 'w8a8')`:\n for policy in (\"fp32\",\"w8a32\",\"w8a8\"):\n # Loop over `batch` in `BATCHES`:\n for batch in BATCHES:\n # Compile and trace the function with XLA (`function`).\n function=jax.jit(lambda x,policy=policy:policy_scores(state,x,calibration,policy))\n # Cast or evaluate `data` in explicit floating-point precision.\n data=export.export(function)(jax.ShapeDtypeStruct((batch,1024),jnp.float32)).serialize()\n # Compute `name` as `f\"{policy}-b{batch}.jaxexport\"`.\n name=f\"{policy}-b{batch}.jaxexport\"\n (path/name).write_bytes(data)\n # Compute deterministic cryptographic digest `artifacts[f'{policy}:{batch}']` for provenance verification.\n artifacts[f\"{policy}:{batch}\"]={\"file\":name,\"sha256\":hashlib.sha256(data).hexdigest()}\n # Convert `manifest` to a host NumPy array for inspection or verification.\n manifest={\"preprocessing\":PREPROCESS,\"classes\":CLASSES,\"batches\":list(BATCHES),\n \"training_data_sha256\":state[\"dataset_sha256\"],\"completed_steps\":state[\"step\"],\n \"parameter_sha256\":digest_arrays(np.asarray(state[\"W\"]),np.asarray(state[\"b\"])),\n \"normalizer_sha256\":digest_arrays(np.asarray(state[\"mean\"]),np.asarray(state[\"std\"])),\n \"calibration_sha256\":calibration[\"calibration_sha256\"],\"jax\":jax.__version__,\n \"activation_scale\":calibration[\"activation_scale\"],\"weight_scales\":calibration[\"weight_scales\"].tolist(),\n \"integer_contract\":{\"weight_axis\":\"output class\",\"zero_point\":0,\"range\":[-127,127],\"bias\":\"float32 after rescaling\",\"codes_dtype\":\"int8\"},\n \"policies\":{\"fp32\":\"FP32 STFT/features/weights/accumulation/output\",\n \"w8a32\":\"FP32 STFT/features; INT8 weight storage dequantized to FP32; FP32 accumulation/bias/output\",\n \"w8a8\":\"FP32 STFT/features; INT8 normalized features and weights; INT32 dot accumulation; FP32 bias/output\"},\n \"native_integer_acceleration_claimed\":False,\"device\":\"CPU\",\"artifacts\":artifacts}\n # Compute `(path/\"release.json\").write_text(json.dumps(manifest,indent` as `2))`.\n (path/\"release.json\").write_text(json.dumps(manifest,indent=2))\n # Return `manifest` to the caller.\n return manifest\n\n# Function `load_release` reference implementation:\ndef load_release(path):\n # Read or serialize artifact data on disk (`manifest`).\n path=Path(path)\n manifest=json.loads((path/\"release.json\").read_text())\n # Guard input contract (`manifest['preprocessing'] != PREPROCESS or manifest['jax'] != jax.__version__`) and fail fast if violated.\n if manifest[\"preprocessing\"]!=PREPROCESS or manifest[\"jax\"]!=jax.__version__:\n raise ValueError(\"release preprocessing/runtime mismatch\")\n # Construct dictionary `artifacts` with the structured fields for this stage.\n artifacts={}\n # Loop over `(key, item)` in `manifest['artifacts'].items()`:\n for key,item in manifest[\"artifacts\"].items():\n # Evaluate the compound expression for `data`.\n data=(path/item[\"file\"]).read_bytes()\n # Guard input contract (`hashlib.sha256(data).hexdigest() != item['sha256']`) and fail fast if violated.\n if hashlib.sha256(data).hexdigest()!=item[\"sha256\"]:raise ValueError(\"artifact checksum mismatch\")\n # Run `export.deserialize` to compute `artifacts[key]`.\n artifacts[key]=export.deserialize(data)\n # Return `(manifest, artifacts)` to the caller.\n return manifest,artifacts" }, { "id": "4", "title": "Measured inference and boundary drills", "command": "python3 projects/audio-harness/tests/check.py --implementation projects/audio-harness/my_model.py --stage 4", "evidence": "Actual CPU timing samples, sample-rate/channel/version failures, silence and explicitly unmeasured edge costs", "expected": "PASS stage 4: measured full preprocessing/inference boundary, malformed request drills and silence behavior", "diagnosis": "Separate window accumulation, decoding, preprocessing, model execution and device measurements before claiming end-to-end latency.", "codeGuide": "In `projects/audio-harness/my_model.py`, implement `infer_release(...)`, `benchmark(...)` using `validate_waveforms`, `contract`, `call`, `jnp.asarray`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 4 verifier.", "starterSnippet": "def infer_release(manifest,artifacts,waveforms,sample_rate=8000,channels=1,\n policy=\"fp32\",version=\"audio-stft-v1\"):\n \"\"\"Stage 4: validated decoded PCM -> completed logits and class_ids.\"\"\"\n # Key APIs to use: `validate_waveforms`, `contract`, `call`, `jnp.asarray`, `block_until_ready`\n # Step 1: Run `validate_waveforms` to compute `a`.\n # Step 2: Guard input contract (`len(a) not in manifest['batches'] or policy not in manifest['policies']`) and fail fast if violated.\n # Step 3: Synchronize host execution until asynchronous device computation completes.\n # Step 4: Return `{'logits': np.asarray(logits), 'class_ids': np.asarray(logits).argmax(axis=1)}` to the caller.\n raise NotImplementedError(\"Reject malformed requests before runtime inference\")\n\ndef benchmark(manifest,artifacts,waveforms,policy=\"fp32\",repeats=30):\n \"\"\"Stage 4: record first/warm timings including validation, placement, exported STFT,\n model completion and host outputs\n exclude capture, buffering, decoding and network.\n \"\"\"\n # Key APIs to use: `time.perf_counter`, `infer_release`, `samples.append`, `np.percentile`\n # Step 1: Record execution timing or profiler trace in `began`.\n # Step 2: Execute the next step of the computation.\n # Step 3: Record execution timing or profiler trace in `first_ms`.\n # Step 4: Evaluate `samples` from the current inputs and state.\n # Step 5: Repeat the update loop over `range(repeats)` steps:\n # Step 6: Inside block: Record execution timing or profiler trace in `began`.\n raise NotImplementedError(\"Measure the actual CPU boundary and report units\")", "solutionSnippet": "# Function `infer_release` reference implementation:\ndef infer_release(manifest,artifacts,waveforms,sample_rate=8000,channels=1,\n policy=\"fp32\",version=\"audio-stft-v1\"):\n # Run `validate_waveforms` to compute `a`.\n a=validate_waveforms(waveforms,sample_rate,channels,version)\n # Guard input contract (`len(a) not in manifest['batches'] or policy not in manifest['policies']`) and fail fast if violated.\n if len(a) not in manifest[\"batches\"] or policy not in manifest[\"policies\"]:\n raise ValueError(\"unsupported batch or precision policy\")\n # Synchronize host execution until asynchronous device computation completes.\n logits=artifacts[f\"{policy}:{len(a)}\"].call(jnp.asarray(a)).block_until_ready()\n # Return `{'logits': np.asarray(logits), 'class_ids': np.asarray(logits).argmax(axis=1)}` to the caller.\n return {\"logits\":np.asarray(logits),\"class_ids\":np.asarray(logits).argmax(axis=1)}\n\n# Function `benchmark` reference implementation:\ndef benchmark(manifest,artifacts,waveforms,policy=\"fp32\",repeats=30):\n # Record execution timing or profiler trace in `began`.\n # Compute `began` as `time.perf_counter()`.\n began=time.perf_counter()\n infer_release(manifest,artifacts,waveforms,policy=policy)\n # Record execution timing or profiler trace in `first_ms`.\n first_ms=(time.perf_counter()-began)*1000\n # Initialize list `samples` for the stage values.\n samples=[]\n # Repeat the update loop over `range(repeats)` steps:\n for _ in range(repeats):\n # Record execution timing or profiler trace in `began`.\n # Compute `began` as `time.perf_counter()`.\n began=time.perf_counter()\n infer_release(manifest,artifacts,waveforms,policy=policy)\n # Record execution timing or profiler trace in ``.\n samples.append((time.perf_counter()-began)*1000)\n # Return `{'first_request_ms': first_ms, 'samples_ms': samples, 'p50_ms': float(np.percentile(samples, 50)), 'p95_ms': float(np.percentile(samples, 95)), 'boundary': 'validated decoded PCM + placement + exported STFT/normalization/classifier + wait + host outputs', 'excluded': 'file decoding, audio capture, window buffering, networking', 'device': 'CPU', 'audio_duration_ms': 128.0, 'examples_per_second': 1000 * len(waveforms) / float(np.percentile(samples, 50))}` to the caller.\n return {\"first_request_ms\":first_ms,\"samples_ms\":samples,\n \"p50_ms\":float(np.percentile(samples,50)),\"p95_ms\":float(np.percentile(samples,95)),\n \"boundary\":\"validated decoded PCM + placement + exported STFT/normalization/classifier + wait + host outputs\",\n \"excluded\":\"file decoding, audio capture, window buffering, networking\",\"device\":\"CPU\",\n \"audio_duration_ms\":128.,\"examples_per_second\":1000*len(waveforms)/float(np.percentile(samples,50))}" } ], "rubric": [ "Explain sample-rate, waveform, crop and STFT contracts with independent calculations", "Preserve recording identities, groups, licenses and split provenance", "Compare a trained spectral classifier with an energy baseline and inspect actual errors", "Recover optimizer, normalization, RNG, crop and iterator state in a fresh process", "Calibrate on training data and distinguish weight, feature, accumulation and output precision", "Reload genuine exports containing spectral preprocessing and reject failed releases", "Measure completed CPU requests while labeling unmeasured capture, network and device costs", "Complete the bundled audio synthesis tasks with changed-condition evidence" ], "workspaceFile": "projects/audio-harness/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/audio-harness/starter/model.py projects/audio-harness/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/audio-harness/starter/model.py projects/audio-harness/my_model.py" } ], "referenceCommand": "python3 projects/audio-harness/tests/check.py --implementation solution --stage all", "setupNote": "Run from the extracted ZIP top folder containing requirements-cpu.txt and projects/audio-harness. The README links executed plots, a connected reference runner and bundled synthesis/reviewer tasks. The fixture classifies synthetic sounds, not speech. Copy the starter scaffold `projects/audio-harness/starter/model.py` to `projects/audio-harness/my_model.py`, implement the stage functions in `projects/audio-harness/my_model.py`, and verify each stage with `python3 projects/audio-harness/tests/check.py --implementation projects/audio-harness/my_model.py --stage 1` (through `--stage 4`)." }, { "schemaVersion": 1, "id": "cross-modal-harness", "title": "Build and audit image–text retrieval", "pathwayId": "models", "status": "authored", "hardware": "CPU synthetic image/text fixture; target runtime unverified", "source": "projects/cross-modal-harness/README.md", "requiredLessonIds": [ "networks-01", "optimization-02", "optimization-09", "transformers-01", "recovery-03", "recovery-04", "deployment-03", "deployment-05" ], "stages": [ { "id": "1", "title": "Pair and preprocessing contracts", "command": "python3 projects/cross-modal-harness/tests/check.py --implementation projects/cross-modal-harness/my_model.py --stage 1", "evidence": "Independent pixel/token checks, stable IDs, source groups and leakage rejection", "expected": "PASS stage 1 plus all earlier stages", "diagnosis": "A shape-correct pair can still be semantically wrong; inspect the actual image and caption.", "codeGuide": "In `projects/cross-modal-harness/my_model.py`, implement `text_features(...)` using `contract`, `representation`, `caption.lower`, `split`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def text_features(captions):\n \"\"\"Exactly one orientation and one thickness; reject missing/ambiguous concepts.\"\"\"\n # Key APIs to use: `contract`, `representation`, `caption.lower`, `split`, `sum`\n # Step 1: Evaluate `result` from the current inputs and state.\n # Step 2: Loop over `caption` in `captions`:\n # Step 3: Inside block: Guard input contract (`not isinstance(caption, str)`) and fail fast if violated.\n # Step 4: Inside block: Run `caption.lower` to compute `words`.\n # Step 5: Guard input contract (`not result`) and fail fast if violated.\n # Step 6: Return `np.asarray(result, np.float32)` to the caller.\n raise NotImplementedError('Implement text_features')", "solutionSnippet": "# Function `text_features` reference implementation:\ndef text_features(captions):\n \"\"\"Exactly one orientation and one thickness; reject missing/ambiguous concepts.\"\"\"\n # Initialize list `result` for the stage values.\n result = []\n # Iterate over `caption` to step through the computation:\n for caption in captions:\n # Guard input contract (`not isinstance(caption, str)`) and fail fast if violated.\n if not isinstance(caption, str):\n raise ValueError('caption must be text')\n # Trace or lower the function to inspect its compiler representation (`words`).\n words = caption.lower().split()\n # Guard input contract (`len(words) != 2 or len(set(words)) != 2`) and fail fast if violated.\n if len(words) != 2 or len(set(words)) != 2:\n raise ValueError('caption needs exactly two different concepts')\n # Guard input contract (`sum((w in words for w in ['vertical', 'horizontal'])) != 1 or sum((w in words for w in ['thin', 'thick'])) != 1`) and fail fast if violated.\n if sum(w in words for w in ['vertical', 'horizontal']) != 1 or sum(w in words for w in ['thin', 'thick']) != 1:\n raise ValueError('caption requires one orientation and one thickness')\n # Guard input contract (`any((w not in CONTRACT['vocabulary'] for w in words))`) and fail fast if violated.\n if any(w not in CONTRACT['vocabulary'] for w in words):\n raise ValueError('unknown concept')\n # Append the current step result to `result`.\n result.append([float(w in words) for w in CONTRACT['vocabulary']])\n # Guard input contract (`not result`) and fail fast if violated.\n if not result:\n raise ValueError('empty caption batch')\n # Return `np.asarray(result, np.float32)` to the caller.\n return np.asarray(result, np.float32)" }, { "id": "2", "title": "Symmetric contrastive objective", "command": "python3 projects/cross-modal-harness/tests/check.py --implementation projects/cross-modal-harness/my_model.py --stage 2", "evidence": "Independent host objective, normalization and finite differences at changed parameters", "expected": "PASS stage 2 plus all earlier stages", "diagnosis": "Inspect the positive mask before adjusting temperature or optimizer settings.", "codeGuide": "In `projects/cross-modal-harness/my_model.py`, implement `image_features(...)` using `np.asarray`, `contract`, `or`, `float32`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def image_features(images):\n \"\"\"Implement the contract from the project guide.\"\"\"\n # Key APIs to use: `np.asarray`, `contract`, `or`, `float32`, `np.isfinite`\n # Step 1: Convert `x` to a host NumPy array for inspection or verification.\n # Step 2: Guard input contract (`x.dtype != np.float32 or x.ndim != 3 or x.shape[1:] != (8, 8) or (not len(x))`) and fail fast if violated.\n # Step 3: Guard input contract (`not np.isfinite(x).all() or np.min(x) < 0 or np.max(x) > 1`) and fail fast if violated.\n # Step 4: Return `x.reshape(len(x), 64)` to the caller.\n raise NotImplementedError('Implement image_features')", "solutionSnippet": "# Function `image_features` reference implementation:\ndef image_features(images):\n # Convert `x` to a host NumPy array for inspection or verification.\n x = np.asarray(images)\n # Guard input contract (`x.dtype != np.float32 or x.ndim != 3 or x.shape[1:] != (8, 8) or (not len(x))`) and fail fast if violated.\n if x.dtype != np.float32 or x.ndim != 3 or x.shape[1:] != (8, 8) or not len(x):\n raise ValueError('images require nonempty float32 (batch,8,8)')\n # Guard input contract (`not np.isfinite(x).all() or np.min(x) < 0 or np.max(x) > 1`) and fail fast if violated.\n if not np.isfinite(x).all() or np.min(x) < 0 or np.max(x) > 1:\n raise ValueError('image range must be finite [0,1]')\n # Return `x.reshape(len(x), 64)` to the caller.\n return x.reshape(len(x), 64)" }, { "id": "3", "title": "Training and complete recovery", "command": "python3 projects/cross-modal-harness/tests/check.py --implementation projects/cross-modal-harness/my_model.py --stage 3", "evidence": "Two training seeds, frozen evaluation and five resumed transitions in a fresh process", "expected": "PASS stage 3 plus all earlier stages", "diagnosis": "Compare pair IDs first, then weights, momentum and the future shuffle key.", "codeGuide": "In `projects/cross-modal-harness/my_model.py`, implement `objective(...)` using `embeddings`, `special.logsumexp`, `jnp.where`, `likelihood`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def objective(params, x, t, labels):\n \"\"\"Implement the contract from the project guide.\"\"\"\n # Key APIs to use: `embeddings`, `special.logsumexp`, `jnp.where`, `likelihood`, `jnp.mean`\n # Step 1: Run `embeddings` to compute `(zi, zt)`.\n # Step 2: Perform matrix contraction / projection to compute `scores`.\n # Step 3: Evaluate `positive` from the current inputs and state.\n # Step 4: Evaluate numerically stable log-space cross-entropy/likelihood (`row`).\n # Step 5: Evaluate numerically stable log-space cross-entropy/likelihood (`col`).\n # Step 6: Return `(jnp.mean(row) + jnp.mean(col)) / 2` to the caller.\n raise NotImplementedError('Implement objective')", "solutionSnippet": "# Function `objective` reference implementation:\ndef objective(params, x, t, labels):\n # Run `embeddings` to compute `(zi, zt)`.\n zi, zt = embeddings(params, x, t)\n # Perform matrix contraction / projection to compute `scores`.\n scores = zi @ zt.T / TRAINING['temperature']\n # Compute `positive` as `labels[:, None] == labels[None, :]`.\n positive = labels[:, None] == labels[None, :]\n # All captions/images of the same semantic class are positives, not false negatives.\n row = jax.scipy.special.logsumexp(scores, axis=1) - jax.scipy.special.logsumexp(jnp.where(positive, scores, -jnp.inf), axis=1)\n # Evaluate numerically stable log-space cross-entropy/likelihood (`col`).\n col = jax.scipy.special.logsumexp(scores, axis=0) - jax.scipy.special.logsumexp(jnp.where(positive, scores, -jnp.inf), axis=0)\n # Return `(jnp.mean(row) + jnp.mean(col)) / 2` to the caller.\n return (jnp.mean(row) + jnp.mean(col)) / 2" }, { "id": "4", "title": "Precision and exported branches", "command": "python3 projects/cross-modal-harness/tests/check.py --implementation projects/cross-modal-harness/my_model.py --stage 4", "evidence": "Calibration provenance, integer oracle and twelve serialized computations loaded in fresh processes", "expected": "PASS stage 4 plus all earlier stages", "diagnosis": "Check preprocessing and scale axes before relaxing parity tolerance.", "codeGuide": "In `projects/cross-modal-harness/my_model.py`, implement `transition(...)` using `contract`, `dataset_hash`, `random.split`, `np.asarray`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 4 verifier.", "starterSnippet": "def transition(state, data):\n \"\"\"Implement the contract from the project guide.\"\"\"\n # Key APIs to use: `contract`, `dataset_hash`, `random.split`, `np.asarray`, `random.permutation`\n # Step 1: Guard input contract (`state['data_hash'] != dataset_hash(data)`) and fail fast if violated.\n # Step 2: Evaluate `(cursor, order, key)` from the current inputs and state.\n # Step 3: Branch on condition `cursor == len(order)`:\n # Step 4: Evaluate `chosen` from the current inputs and state.\n # Step 5: Run `image_features` to compute `x`.\n # Step 6: Run `text_features` to compute `t`.\n raise NotImplementedError('Implement transition')", "solutionSnippet": "# Function `transition` reference implementation:\ndef transition(state, data):\n # Guard input contract (`state['data_hash'] != dataset_hash(data)`) and fail fast if violated.\n if state['data_hash'] != dataset_hash(data):\n raise ValueError('data identity changed')\n # Compute `cursor, order, key` as `state['cursor'], state['order'], state['key']`.\n cursor, order, key = state['cursor'], state['order'], state['key']\n # Branch on condition `cursor == len(order)`:\n if cursor == len(order):\n key, shuffle = jax.random.split(key)\n order = np.asarray(jax.random.permutation(shuffle, len(order)))\n cursor = 0\n # Compute `chosen` as `order[cursor:cursor + state['batch_size']]`.\n chosen = order[cursor:cursor + state['batch_size']]\n # Run `image_features` to compute `x`.\n # Run `text_features` to compute `t`.\n x = image_features(data['images'][chosen])\n t = text_features([data['captions'][i] for i in chosen])\n # Create device-backed JAX array `(params, momentum, loss)`.\n params, momentum, loss = _update(state['params'], state['momentum'], jnp.asarray(x), jnp.asarray(t), jnp.asarray(data['labels'][chosen]))\n # Synchronize host execution until asynchronous device computation completes.\n loss.block_until_ready()\n # Construct dictionary `new` with the structured fields for this stage.\n new = {**state, 'params': params, 'momentum': momentum, 'key': key, 'order': order,\n 'cursor': cursor + len(chosen), 'step': state['step'] + 1}\n # Return `(new, {'loss': float(loss), 'ids': [data['ids'][i] for i in chosen], 'count': len(chosen)})` to the caller.\n return new, {'loss': float(loss), 'ids': [data['ids'][i] for i in chosen], 'count': len(chosen)}" }, { "id": "5", "title": "Inference, measured release and rollback", "command": "python3 projects/cross-modal-harness/tests/check.py --implementation projects/cross-modal-harness/my_model.py --stage 5", "evidence": "Boundary rejection, raw timings, degraded-candidate rejection and pointer preservation", "expected": "PASS stage 5 plus all earlier stages", "diagnosis": "A good file checksum is verified separately from model quality; evaluate the candidate before selecting it.", "codeGuide": "In `projects/cross-modal-harness/my_model.py`, implement `load_checkpoint(...)`, `encoder(...)`, `infer(...)`, `activate(...)` using `disk`, `json.loads`, `Path`, `read_text`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 5 verifier.", "starterSnippet": "def load_checkpoint(folder, data, batch_size=16):\n \"\"\"Implement the contract from the project guide.\"\"\"\n # Key APIs to use: `disk`, `json.loads`, `Path`, `read_text`, `contract`\n # Step 1: Read or serialize artifact data on disk (`record`).\n # Step 2: Evaluate `p` from the current inputs and state.\n # Step 3: Evaluate `s` from the current inputs and state.\n # Step 4: Guard input contract (`digest(p) != record['sha256'] or p['contract'] != CONTRACT or p.get('training') != TRAINING or (p.get('implementation_hash') != implementation_hash()) or (s['data_hash'] != dataset_hash(data)) or (s['batch_size'] != batch_size)`) and fail fast if violated.\n # Step 5: Guard input contract (`sorted(s['order']) != list(range(len(data['ids']))) or not 0 <= s['cursor'] <= len(s['order']) or s['step'] < 0`) and fail fast if violated.\n # Step 6: Guard input contract (`len(s['key']) != 2`) and fail fast if violated.\n raise NotImplementedError('Implement load_checkpoint')\n\ndef encoder(params, branch, policy, calibration):\n \"\"\"Implement the contract from the project guide.\"\"\"\n # Key APIs to use: `contract`, `jnp.asarray`, `quantize_weights`, `normalize`, `np.isfinite`\n # Step 1: Guard input contract (`branch not in ['image', 'text'] or policy not in ['fp32', 'w8a32', 'w8a8']`) and fail fast if violated.\n # Step 2: Create device-backed JAX array `w`.\n # Step 3: Run `quantize_weights` to compute `(qw, scale)`.\n # Step 4: Branch on condition `policy == 'fp32'`:\n # Step 5: Branch on condition `policy == 'w8a32'`:\n # Step 6: Evaluate `activation_scale` from the current inputs and state.\n raise NotImplementedError('Implement encoder')\n\ndef infer(artifact, *, images=None, captions=None):\n \"\"\"Implement the contract from the project guide.\"\"\"\n # Key APIs to use: `contract`, `if`, `image_features`, `text_features`, `call`\n # Step 1: Guard input contract (`(images is None) == (captions is None)`) and fail fast if violated.\n # Step 2: Evaluate `branch` from the current inputs and state.\n # Step 3: Run `image_features` to compute `x`.\n # Step 4: Evaluate `name` from the current inputs and state.\n # Step 5: Guard input contract (`name not in artifact['models']`) and fail fast if violated.\n # Step 6: Create device-backed JAX array `result`.\n raise NotImplementedError('Implement infer')\n\ndef activate(registry, candidate, evaluation_images, evaluation_labels, minimum_accuracy=0.75):\n \"\"\"Measured fixture quality gate before an atomic local pointer change.\"\"\"\n # Key APIs to use: `contract`, `load_artifact`, `np.asarray`, `infer`, `np.arange`\n # Step 1: Guard input contract (`not 0 <= minimum_accuracy <= 1`) and fail fast if violated.\n # Step 2: Run `load_artifact` to compute `artifact`.\n # Step 3: Guard input contract (`len(evaluation_images) != 8 or np.asarray(evaluation_labels).shape != (8,)`) and fail fast if violated.\n # Step 4: Evaluate `bank` from the current inputs and state.\n # Step 5: Run `infer` to compute `zi`.\n # Step 6: Run `infer` to compute `zt`.\n raise NotImplementedError('Implement activate')", "solutionSnippet": "# Function `load_checkpoint` reference implementation:\ndef load_checkpoint(folder, data, batch_size=16):\n # Read or serialize artifact data on disk (`record`).\n record = json.loads((Path(folder)/'checkpoint.json').read_text())\n # Compute `p` as `record['payload']`.\n p = record['payload']\n s = p['state']\n # Guard input contract (`digest(p) != record['sha256'] or p['contract'] != CONTRACT or p.get('training') != TRAINING or (p.get('implementation_hash') != implementation_hash()) or (s['data_hash'] != dataset_hash(data)) or (s['batch_size'] != batch_size)`) and fail fast if violated.\n if digest(p) != record['sha256'] or p['contract'] != CONTRACT or p.get('training') != TRAINING or p.get('implementation_hash') != implementation_hash() or s['data_hash'] != dataset_hash(data) or s['batch_size'] != batch_size:\n raise ValueError('checkpoint identity, configuration or checksum mismatch')\n # Guard input contract (`sorted(s['order']) != list(range(len(data['ids']))) or not 0 <= s['cursor'] <= len(s['order']) or s['step'] < 0`) and fail fast if violated.\n if sorted(s['order']) != list(range(len(data['ids']))) or not 0 <= s['cursor'] <= len(s['order']) or s['step'] < 0:\n raise ValueError('invalid iterator state')\n # Guard input contract (`len(s['key']) != 2`) and fail fast if violated.\n if len(s['key']) != 2:\n raise ValueError('invalid random state')\n # Loop over `group` in `['params', 'momentum']`:\n for group in ['params', 'momentum']:\n # Loop over `(k, shape)` in `[('image', (64, 4)), ('text', (4, 4))]`:\n for k, shape in [('image', (64,4)), ('text', (4,4))]:\n # Convert `v` to a host NumPy array for inspection or verification.\n v = np.asarray(s[group][k], np.float32)\n # Guard input contract (`v.shape != shape or not np.isfinite(v).all()`) and fail fast if violated.\n if v.shape != shape or not np.isfinite(v).all():\n raise ValueError('invalid parameter state')\n # Create device-backed JAX array `s[group][k]`.\n s[group][k] = jnp.asarray(v)\n # Create device-backed JAX array `s['key']`.\n # Convert `s['order']` to a host NumPy array for inspection or verification.\n s['key'] = jnp.asarray(s['key'], jnp.uint32)\n s['order'] = np.asarray(s['order'], np.int32)\n # Return `s` to the caller.\n return s\n\n# Function `encoder` reference implementation:\ndef encoder(params, branch, policy, calibration):\n # Guard input contract (`branch not in ['image', 'text'] or policy not in ['fp32', 'w8a32', 'w8a8']`) and fail fast if violated.\n if branch not in ['image', 'text'] or policy not in ['fp32','w8a32','w8a8']:\n raise ValueError('unsupported branch or precision')\n # Create device-backed JAX array `w`.\n # Run `quantize_weights` to compute `(qw, scale)`.\n w = jnp.asarray(params[branch])\n qw, scale = quantize_weights(w)\n # Branch on condition `policy == 'fp32'`:\n if policy == 'fp32': return lambda x: normalize(x @ w)\n # Branch on condition `policy == 'w8a32'`:\n if policy == 'w8a32': return lambda x: normalize(x @ (jnp.asarray(qw, jnp.float32) * jnp.asarray(scale)))\n # Compute `activation_scale` as `calibration['input_scales'][branch]`.\n activation_scale = calibration['input_scales'][branch]\n # Guard input contract (`not np.isfinite(activation_scale) or activation_scale <= 0`) and fail fast if violated.\n if not np.isfinite(activation_scale) or activation_scale <= 0:\n raise ValueError('invalid calibration')\n # Function `run(x)` implementing this stage's computation:\n def run(x):\n # Combine or mask array elements to form `qx`.\n qx = jnp.clip(jnp.rint(x / activation_scale), -127, 127).astype(jnp.int8)\n # Create device-backed JAX array `accumulation`.\n accumulation = qx.astype(jnp.int32) @ jnp.asarray(qw, jnp.int32)\n # Return `normalize(accumulation.astype(jnp.float32) * activation_scale * jnp.asarray(scale))` to the caller.\n return normalize(accumulation.astype(jnp.float32) * activation_scale * jnp.asarray(scale))\n # Return `run` to the caller.\n return run\n\n# Function `infer` reference implementation:\ndef infer(artifact, *, images=None, captions=None):\n # Guard input contract (`(images is None) == (captions is None)`) and fail fast if violated.\n if (images is None) == (captions is None):\n raise ValueError('supply exactly one modality per embedding request')\n # Compute `branch` as `'image' if images is not None else 'text'`.\n branch = 'image' if images is not None else 'text'\n # Run `image_features` to compute `x`.\n x = image_features(images) if images is not None else text_features(captions)\n # Compute `name` as `f'{branch}-{len(x)}.jax'`.\n name = f'{branch}-{len(x)}.jax'\n # Guard input contract (`name not in artifact['models']`) and fail fast if violated.\n if name not in artifact['models']:\n raise ValueError('supported batch sizes are 1 and 8')\n # Create device-backed JAX array `result`.\n result = artifact['models'][name].call(jnp.asarray(x))\n # Synchronize host execution until asynchronous device computation completes.\n result.block_until_ready()\n # Return `np.asarray(result)` to the caller.\n return np.asarray(result)\n\n# Function `activate` reference implementation:\ndef activate(registry, candidate, evaluation_images, evaluation_labels, minimum_accuracy=0.75):\n \"\"\"Measured fixture quality gate before an atomic local pointer change.\"\"\"\n # Guard input contract (`not 0 <= minimum_accuracy <= 1`) and fail fast if violated.\n if not 0 <= minimum_accuracy <= 1:\n raise ValueError('invalid quality threshold')\n # Run `load_artifact` to compute `artifact`.\n artifact = load_artifact(candidate)\n # Guard input contract (`len(evaluation_images) != 8 or np.asarray(evaluation_labels).shape != (8,)`) and fail fast if violated.\n if len(evaluation_images) != 8 or np.asarray(evaluation_labels).shape != (8,):\n raise ValueError('quality gate needs eight declared labeled examples')\n # Initialize list `bank` for the stage values.\n bank = ['vertical thin','vertical thick','horizontal thin','horizontal thick'] * 2\n # Run `infer` to compute `zi`.\n # Run `infer` to compute `zt`.\n zi = infer(artifact, images=evaluation_images)\n zt = infer(artifact, captions=bank)\n # Create evenly spaced index values in `predicted`.\n predicted = np.arange(8)[(zi @ zt.T).argmax(axis=1)] % 4\n # Reduce across the target axis to summarize `score`.\n score = float(np.mean(predicted == evaluation_labels))\n # Guard input contract (`score < minimum_accuracy`) and fail fast if violated.\n if score < minimum_accuracy:\n raise ValueError('candidate failed measured retrieval gate')\n # Compute `registry` as `Path(registry)`.\n registry=Path(registry)\n registry.mkdir(parents=True,exist_ok=True)\n # Read or serialize artifact data on disk (`record`).\n record={'folder':str(Path(candidate).resolve()),'manifest_sha256':hashlib.sha256((Path(candidate)/'manifest.json').read_bytes()).hexdigest(),'gate_accuracy':score}\n # Compute `temporary` as `registry/'ACTIVE.tmp'`.\n temporary=registry/'ACTIVE.tmp'\n temporary.write_text(json.dumps(record))\n temporary.replace(registry/'ACTIVE.json')\n # Return `record` to the caller.\n return record" } ], "rubric": [ "Validate paired semantic inputs and split by source identity", "Derive and independently check a multi-positive objective", "Train both encoders and inspect clean versus shifted retrieval", "Restore optimizer, random and iterator state in a fresh process", "Verify calibrated integer arithmetic and serialized branch parity", "Measure the actual request boundary and reject a degraded release", "Bound conclusions to the synthetic CPU fixture" ], "workspaceFile": "projects/cross-modal-harness/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/cross-modal-harness/starter/model.py projects/cross-modal-harness/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/cross-modal-harness/starter/model.py projects/cross-modal-harness/my_model.py" } ], "referenceCommand": "python3 projects/cross-modal-harness/tests/check.py --implementation solution --stage all", "setupNote": "Extract the ZIP and install its requirements-cpu.txt; run commands from the extracted top folder. Reference figures are measured on a synthetic CPU fixture. Copy the starter scaffold `projects/cross-modal-harness/starter/model.py` to `projects/cross-modal-harness/my_model.py`, implement the stage functions in `projects/cross-modal-harness/my_model.py`, and verify each stage with `python3 projects/cross-modal-harness/tests/check.py --implementation projects/cross-modal-harness/my_model.py --stage 1` (through `--stage 5`)." }, { "schemaVersion": 1, "id": "kernel-audit", "title": "Audit a Pallas kernel and its target evidence", "pathwayId": "scale", "status": "authored", "hardware": "CPU interpretation/simulation available; real TPU runner requires separately validated TPU hardware", "source": "projects/kernel-audit/README.md", "requiredLessonIds": [ "performance-01", "performance-02", "distributed-01", "distributed-02", "kernels-01", "kernels-02", "kernels-03", "kernels-04" ], "stages": [ { "id": "1", "title": "Grid and precision contract", "command": "python3 projects/kernel-audit/tests/check.py --implementation projects/kernel-audit/my_model.py --stage 1", "evidence": "Independent full-array checks, singleton/tail fixtures, block ownership and rejected input contracts", "expected": "PASS stage 1: actual grid semantics, full and partial tiles, singleton rows, two dtypes and rejected contracts", "diagnosis": "If the last row or column fails, check ceiling division, padding and crop boundaries before changing arithmetic.", "codeGuide": "In `projects/kernel-audit/my_model.py`, implement `blocked_axpy(...)` using `pad_pair`, `specification`, `pl.BlockSpec`, `pl.pallas_call`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def blocked_axpy(x,y,block=(2,4)):\n \"\"\"Actual interpreted Pallas Z=2X+Y, matching nonempty matrix shapes/dtypes.\n\n Support float32/bfloat16 with float32 arithmetic and one output cast.\n Validate positive blocks, pad full tiles explicitly, crop logical output.\n \"\"\"\n # Key APIs to use: `pad_pair`, `specification`, `pl.BlockSpec`, `pl.pallas_call`, `jax.ShapeDtypeStruct`\n # Step 1: Combine or mask array elements to form `(px, py, padded)`.\n # Step 2: Evaluate `(bm, bn)` from the current inputs and state.\n # Step 3: Invoke custom Pallas kernel or tile specification (`spec`).\n # Step 4: Combine or mask array elements to form `out`.\n # Step 5: Return `out[:x.shape[0], :x.shape[1]]` to the caller.\n raise NotImplementedError('Stage 1: grid, BlockSpecs and explicit tails')", "solutionSnippet": "# Function `blocked_axpy` reference implementation:\ndef blocked_axpy(x,y,block=(2,4)):\n # Combine or mask array elements to form `(px, py, padded)`.\n px,py,padded=pad_pair(x,y,block)\n # Compute `bm,bn` as `block`.\n bm,bn=block\n # Invoke custom Pallas kernel or tile specification (`spec`).\n spec=pl.BlockSpec(block,lambda i,j:(i,j))\n # Combine or mask array elements to form `out`.\n out=pl.pallas_call(axpy_body,\n out_shape=jax.ShapeDtypeStruct(padded,x.dtype),\n grid=(padded[0]//bm,padded[1]//bn),\n in_specs=(spec,spec),out_specs=spec,interpret=True)(px,py)\n # Return `out[:x.shape[0], :x.shape[1]]` to the caller.\n return out[:x.shape[0],:x.shape[1]]" }, { "id": "2", "title": "Fused target-oriented kernel", "command": "python3 projects/kernel-audit/tests/check.py --implementation projects/kernel-audit/my_model.py --stage 2", "evidence": "Bias-axis and activation-boundary checks, two precisions and an explicit no-fallback target guard", "expected": "PASS stage 2: fused broadcast contract, ReLU boundaries, tails, two precisions and no CPU target fallback", "diagnosis": "If interpretation passes but target lowering fails, preserve the target error; interpretation is not a target compilation test.", "codeGuide": "In `projects/kernel-audit/my_model.py`, implement `fused_bias_relu(...)` using `contract`, `or`, `min`, `be`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def fused_bias_relu(x,bias,block=(8,128),mode='interpret'):\n \"\"\"Actual Pallas max(X+bias,0), matrix X and vector bias of matching dtype.\n\n interpret is CPU semantics\n tpu must refuse non-TPU backend/inputs and use\n interpret=False. Target blocks are multiples of (8,128). Cast arithmetic to\n float32 and output once to the input dtype. Preserve logical output shape.\n \"\"\"\n # Key APIs to use: `contract`, `or`, `min`, `be`, `in`\n # Step 1: Guard input contract (`x.ndim != 2 or bias.ndim != 1 or bias.shape[0] != x.shape[1] or (min(x.shape) < 1)`) and fail fast if violated.\n # Step 2: Guard input contract (`x.dtype != bias.dtype or x.dtype not in (jnp.float32, jnp.bfloat16)`) and fail fast if violated.\n # Step 3: Evaluate `(bm, bn)` from the current inputs and state.\n # Step 4: Guard input contract (`not isinstance(bm, int) or not isinstance(bn, int) or bm < 1 or (bn < 1)`) and fail fast if violated.\n # Step 5: Guard input contract (`mode not in ('interpret', 'tpu')`) and fail fast if violated.\n # Step 6: Branch on condition `mode == 'tpu'`:\n raise NotImplementedError('Stage 2: fused target path with no CPU fallback')", "solutionSnippet": "# Function `fused_bias_relu` reference implementation:\ndef fused_bias_relu(x,bias,block=(8,128),mode=\"interpret\"):\n # Guard input contract (`x.ndim != 2 or bias.ndim != 1 or bias.shape[0] != x.shape[1] or (min(x.shape) < 1)`) and fail fast if violated.\n if x.ndim!=2 or bias.ndim!=1 or bias.shape[0]!=x.shape[1] or min(x.shape)<1:\n raise ValueError(\"x must be (M,N), bias must be (N,), and dimensions nonempty\")\n # Guard input contract (`x.dtype != bias.dtype or x.dtype not in (jnp.float32, jnp.bfloat16)`) and fail fast if violated.\n if x.dtype!=bias.dtype or x.dtype not in (jnp.float32,jnp.bfloat16):\n raise ValueError(\"matching float32 or bfloat16 required\")\n # Compute `bm,bn` as `block`.\n bm,bn=block\n # Guard input contract (`not isinstance(bm, int) or not isinstance(bn, int) or bm < 1 or (bn < 1)`) and fail fast if violated.\n if not isinstance(bm,int) or not isinstance(bn,int) or bm<1 or bn<1:\n raise ValueError(\"positive integer block sizes required\")\n # Guard input contract (`mode not in ('interpret', 'tpu')`) and fail fast if violated.\n if mode not in (\"interpret\",\"tpu\"):\n raise ValueError(\"mode must explicitly be interpret or tpu\")\n # Branch on condition `mode == 'tpu'`:\n if mode==\"tpu\":\n if jax.default_backend()!=\"tpu\" or any(not isinstance(a,jax.core.Tracer) and any(d.platform!=\"tpu\" for d in a.devices()) for a in (x,bias)):\n raise RuntimeError(\"Real TPU inputs/backend required; CPU fallback is disabled\")\n if bm%8 or bn%128:\n raise ValueError(\"this TPU wrapper requires block multiples of (8,128)\")\n # Compute `m,n` as `x.shape`.\n m,n=x.shape\n pm=(m+bm-1)//bm*bm\n pn=(n+bn-1)//bn*bn\n # Combine or mask array elements to form `px`.\n px=jnp.pad(x,((0,pm-m),(0,pn-n)))\n # Combine or mask array elements to form `pb`.\n pb=jnp.pad(bias,(0,pn-n))[None,:]\n # Function `kernel(x_ref, bias_ref, out_ref)` implementing this stage's computation:\n def kernel(x_ref,bias_ref,out_ref):\n # Cast or evaluate `value` in explicit floating-point precision.\n value=x_ref[...].astype(jnp.float32)+bias_ref[...].astype(jnp.float32)\n # Reduce across the target axis to summarize `out_ref[...]`.\n out_ref[...]=jnp.maximum(value,0).astype(out_ref.dtype)\n # Invoke custom Pallas kernel or tile specification (`matrix_spec`).\n matrix_spec=pl.BlockSpec((bm,bn),lambda i,j:(i,j))\n # Invoke custom Pallas kernel or tile specification (`bias_spec`).\n bias_spec=pl.BlockSpec((1,bn),lambda i,j:(0,j))\n # Invoke custom Pallas kernel or tile specification (`output`).\n output=pl.pallas_call(kernel,\n out_shape=jax.ShapeDtypeStruct((pm,pn),x.dtype),\n grid=(pm//bm,pn//bn),in_specs=(matrix_spec,bias_spec),out_specs=matrix_spec,\n interpret=(mode==\"interpret\"),\n compiler_params=pltpu.CompilerParams(dimension_semantics=(\"parallel\",\"parallel\")))(px,pb)\n # Return `output[:m, :n]` to the caller.\n return output[:m,:n]" }, { "id": "3", "title": "Pipeline semantics and measurement protocol", "command": "python3 projects/kernel-audit/tests/check.py --implementation projects/kernel-audit/my_model.py --stage 3", "evidence": "Synchronous/buffered comparisons, two/three input buffers, representative shapes and guarded target benchmark; target receipts remain separate", "expected": "PASS stage 3: actual pipeline CPU simulation, synchronous/buffered agreement, input buffering, tails/dtypes and guarded target benchmark", "diagnosis": "If buffering changes values, inspect copies and ownership. If hardware is unavailable, keep performance unverified rather than timing simulation.", "codeGuide": "In `projects/kernel-audit/my_model.py`, implement `pipelined_axpy(...)`, `target_benchmark(...)` using `pad_pair`, `contract`, `of`, `in`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def pipelined_axpy(x,y,block=(8,128),buffers=2,no_pipelining=False,mode='simulate'):\n \"\"\"Actual emit_pipeline for Z=2X+Y with explicit tails and dtype policy.\n\n Two or three input buffers\n two output buffers. Blocks multiple of (8,128).\n simulate uses TPU InterpretParams and explicitly simulated layout metadata;\n tpu requires real TPU backend/inputs and interpret=False. Never substitute\n simulation for a target request. no_pipelining exposes synchronous debug\n copies. Check representative shapes before discussing performance.\n \"\"\"\n # Key APIs to use: `pad_pair`, `contract`, `of`, `in`, `and`\n # Step 1: Combine or mask array elements to form `(px, py, padded)`.\n # Step 2: Evaluate `(bm, bn)` from the current inputs and state.\n # Step 3: Guard input contract (`bm % 8 or bn % 128`) and fail fast if violated.\n # Step 4: Guard input contract (`buffers not in (2, 3)`) and fail fast if violated.\n # Step 5: Guard input contract (`mode not in ('simulate', 'tpu')`) and fail fast if violated.\n # Step 6: Guard input contract (`mode == 'tpu' and (jax.default_backend() != 'tpu' or any((not isinstance(a, jax.core.Tracer) and any((d.platform != 'tpu' for d in a.devices())) for a in (x, y))))`) and fail fast if violated.\n raise NotImplementedError('Stage 3: pipeline semantics and evidence boundary')\n\ndef target_benchmark(candidate,baseline,args,repeats=20):\n \"\"\"Refuse non-TPU inputs/backend; compile separately, warm up, synchronize.\n\n Return actual_backend, device_kind, boundary and candidate/baseline records\n containing compile_seconds, samples_ms, median_ms and p90_ms. Time the full\n logical-input padded/cropped wrapper, excluding input placement/compilation.\n Require at least five samples. CPU interpretation is not timed here.\n \"\"\"\n # Key APIs to use: `contract`, `jax.default_backend`, `any`, `a.devices`, `in`\n # Step 1: Guard input contract (`jax.default_backend() != 'tpu' or any((d.platform != 'tpu' for a in args for d in a.devices()))`) and fail fast if violated.\n # Step 2: Guard input contract (`repeats < 5`) and fail fast if violated.\n # Step 3: Import required JAX, NumPy, and standard-library modules.\n # Step 4: Evaluate `records` from the current inputs and state.\n # Step 5: Loop over `(label, function)` in `(('candidate', candidate), ('baseline', baseline))`:\n # Step 6: Inside block: Record execution timing or profiler trace in `started`.\n raise NotImplementedError('Stage 3: honest target-only measurement protocol')", "solutionSnippet": "# Function `pipelined_axpy` reference implementation:\ndef pipelined_axpy(x,y,block=(8,128),buffers=2,no_pipelining=False,mode=\"simulate\"):\n # Combine or mask array elements to form `(px, py, padded)`.\n px,py,padded=pad_pair(x,y,block)\n # Compute `bm,bn` as `block`.\n bm,bn=block\n # Guard input contract (`bm % 8 or bn % 128`) and fail fast if violated.\n if bm%8 or bn%128:\n raise ValueError(\"pipeline blocks must be multiples of (8,128)\")\n # Guard input contract (`buffers not in (2, 3)`) and fail fast if violated.\n if buffers not in (2,3):\n raise ValueError(\"this lab supports two or three buffers\")\n # Guard input contract (`mode not in ('simulate', 'tpu')`) and fail fast if violated.\n if mode not in (\"simulate\",\"tpu\"):\n raise ValueError(\"mode must explicitly be simulate or tpu\")\n # Guard input contract (`mode == 'tpu' and (jax.default_backend() != 'tpu' or any((not isinstance(a, jax.core.Tracer) and any((d.platform != 'tpu' for d in a.devices())) for a in (x, y))))`) and fail fast if violated.\n if mode==\"tpu\" and (jax.default_backend()!=\"tpu\" or any(not isinstance(a,jax.core.Tracer) and any(d.platform!=\"tpu\" for d in a.devices()) for a in (x,y))):\n raise RuntimeError(\"Real TPU inputs/backend required; simulation fallback is disabled\")\n # Invoke custom Pallas kernel or tile specification (`spec`).\n spec=pl.BlockSpec(block,lambda i,j:(i,j),pipeline_mode=pl.Buffered(buffer_count=buffers))\n # Invoke custom Pallas kernel or tile specification (`output_spec`).\n output_spec=pl.BlockSpec(block,lambda i,j:(i,j),pipeline_mode=pl.Buffered(buffer_count=2))\n # Function `outer(x_hbm, y_hbm, out_hbm)` implementing this stage's computation:\n def outer(x_hbm,y_hbm,out_hbm):\n # Combine or mask array elements to form ``.\n pltpu.emit_pipeline(axpy_body,grid=(padded[0]//bm,padded[1]//bn),\n in_specs=(spec,spec),out_specs=output_spec,\n no_pipelining=no_pipelining)(x_hbm,y_hbm,out_hbm)\n # Invoke custom Pallas kernel or tile specification (`whole`).\n whole=pl.BlockSpec(memory_space=pl.ANY)\n # Run `pltpu.InterpretParams` to compute `interpretation`.\n interpretation=pltpu.InterpretParams(detect_races=True) if mode==\"simulate\" else False\n # Combine or mask array elements to form `call`.\n call=pl.pallas_call(outer,out_shape=jax.ShapeDtypeStruct(padded,x.dtype),\n in_specs=(whole,whole),out_specs=whole,interpret=interpretation)\n # Branch on condition `mode == 'simulate'`:\n if mode==\"simulate\":\n # This describes a SIMULATED TPU layout, not the machine running the code.\n abstract=jax.sharding.AbstractMesh((1,),(\"simulated_device\",),\n abstract_device=jax.sharding.AbstractDevice(device_kind=\"TPU v5 lite\",num_cores=1))\n with jax.sharding.use_abstract_mesh(abstract):\n output=call(px,py)\n else:\n output=call(px,py)\n # Return `output[:x.shape[0], :x.shape[1]]` to the caller.\n return output[:x.shape[0],:x.shape[1]]\n\n# Function `target_benchmark` reference implementation:\ndef target_benchmark(candidate,baseline,args,repeats=20):\n # Guard input contract (`jax.default_backend() != 'tpu' or any((d.platform != 'tpu' for a in args for d in a.devices()))`) and fail fast if violated.\n if jax.default_backend()!=\"tpu\" or any(d.platform!=\"tpu\" for a in args for d in a.devices()):\n raise RuntimeError(\"TPU benchmark requires real TPU inputs; interpretation timings are not accepted\")\n # Guard input contract (`repeats < 5`) and fail fast if violated.\n if repeats<5:\n raise ValueError(\"at least five repeated target measurements required\")\n # Import time for this computation.\n import time\n # Construct dictionary `records` with the structured fields for this stage.\n records={}\n # Iterate over `(label, function)` to step through the computation:\n for label,function in ((\"candidate\",candidate),(\"baseline\",baseline)):\n # Record execution timing or profiler trace in `started`.\n started=time.perf_counter()\n # Wrap with `jax.jit` (`compiled`) so XLA traces and compiles the function.\n compiled=jax.jit(function).lower(*args).compile()\n # Record execution timing or profiler trace in `compile_seconds`.\n compile_seconds=time.perf_counter()-started\n # Repeat the update loop over `range(5)` steps:\n # Synchronize host execution until asynchronous device computation completes.\n for _ in range(5):compiled(*args).block_until_ready()\n # Initialize list `samples` for the stage values.\n samples=[]\n # Repeat the update loop over `range(repeats)` steps:\n for _ in range(repeats):\n # Record execution timing or profiler trace in `started`.\n started=time.perf_counter()\n # Synchronize host execution until asynchronous device computation completes.\n compiled(*args).block_until_ready()\n # Record execution timing or profiler trace in ``.\n samples.append((time.perf_counter()-started)*1000)\n # Construct dictionary `records[label]` with the structured fields for this stage.\n records[label]={\"compile_seconds\":compile_seconds,\"samples_ms\":samples,\n \"median_ms\":float(np.median(samples)),\"p90_ms\":float(np.percentile(samples,90))}\n # Check which hardware backend (`cpu`, `gpu`, or `tpu`) JAX selected for `records['actual_backend']`.\n records[\"actual_backend\"]=jax.default_backend()\n # Query the active JAX devices into `records['device_kind']`.\n records[\"device_kind\"]=jax.devices()[0].device_kind\n # Combine or mask array elements to form `records['boundary']`.\n records[\"boundary\"]=\"already-placed logical inputs; complete padded/cropped wrapper output; warm synchronized execution\"\n # Return `records` to the caller.\n return records" } ], "rubric": [ "Verify every output against an independent represented-input oracle across tails and dtypes", "Explain grid coordinates, input windows and unique output ownership", "Preserve an explicit float32 compute and final output-cast policy", "Execute actual synchronous and buffered Pallas pipeline semantics", "Reject unsupported shape, dtype, buffer and target-mode contracts", "Separate CPU simulation metadata from actual target backend evidence", "Use a guarded TPU-only runner with synchronized warm measurements and a strong JAX baseline", "Relate profiling and partitioning decisions to the end-to-end optimization opportunity" ], "workspaceFile": "projects/kernel-audit/my_model.py", "prepare": [ { "label": "macOS / Linux", "command": "cp projects/kernel-audit/starter/model.py projects/kernel-audit/my_model.py" }, { "label": "Windows PowerShell", "command": "Copy-Item projects/kernel-audit/starter/model.py projects/kernel-audit/my_model.py" } ], "referenceCommand": "python3 projects/kernel-audit/tests/check.py --implementation solution --stage all", "setupNote": "CPU stages use requirements-cpu.txt with actual Pallas interpretation and TPU-memory simulation. Run from the extracted top folder. The separate target_tpu.py command requires a configured real TPU environment and refuses CPU fallback; no TPU qualification is implied by CPU stage passes. Copy the starter scaffold `projects/kernel-audit/starter/model.py` to `projects/kernel-audit/my_model.py`, implement the stage functions in `projects/kernel-audit/my_model.py`, and verify each stage with `python3 projects/kernel-audit/tests/check.py --implementation projects/kernel-audit/my_model.py --stage 1` (through `--stage 3`).", "assessmentId": "scale" }, { "schemaVersion": 1, "id": "image-harness", "title": "Build a recoverable image classifier through exported inference", "pathwayId": "models", "status": "authored", "hardware": "CPU; target-device extension requires separate validation", "source": "projects/image-harness/README.md", "requiredLessonIds": [ "networks-01", "networks-02", "networks-03", "networks-04", "networks-05", "recovery-01", "recovery-02", "recovery-03", "recovery-04", "deployment-01", "deployment-03", "deployment-05", "deployment-06", "performance-01", "performance-02" ], "stages": [ { "id": "1", "title": "Image data and preprocessing", "command": "python3 projects/image-harness/tests/check.py --stage 1 --implementation projects/image-harness/my_model.py", "evidence": "Channel/range/layout references, PNG ingestion, source-group split manifest", "expected": "PASS stage 1", "diagnosis": "Reject ambiguous input ranges and review source groups before fitting.", "codeGuide": "In `projects/image-harness/my_model.py`, implement `validate_splits(...)`, `preprocess(...)` using `in`, `contract`, `np.asarray`, `np.issubdtype`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 1 verifier.", "starterSnippet": "def validate_splits(training,heldout):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `in`, `contract`, `np.asarray`, `np.issubdtype`, `np.any`\n # Step 1: Loop over `data` in `(training, heldout)`:\n # Step 2: Inside block: Run `len` to compute `n`.\n # Step 3: Inside block: Guard input contract (`np.asarray(data['labels']).shape != (n,) or len(data['ids']) != n or len(data['groups']) != n`) and fail fast if violated.\n # Step 4: Guard input contract (`set(training['ids']) & set(heldout['ids']) or set(training['groups']) & set(heldout['groups'])`) and fail fast if violated.\n # Step 5: Evaluate `train_pixels` from the current inputs and state.\n # Step 6: Guard input contract (`train_pixels & {digest_arrays(image) for image in heldout['images']}`) and fail fast if violated.\n raise NotImplementedError('Implement validate_splits')\n\ndef preprocess(images,layout='NHWC',input_range='uint8'):\n 'Convert declared layouts/ranges to (B,8,8,1) float32.'\n # Key APIs to use: `np.asarray`, `contract`, `np.transpose`, `in`, `min`\n # Step 1: Convert `a` to a host NumPy array for inspection or verification.\n # Step 2: Guard input contract (`a.ndim != 4 or not len(a)`) and fail fast if violated.\n # Step 3: Branch on condition `layout == 'NCHW'`:\n # Step 4: Guard input contract (`a.shape[-1] not in (1, 3) or min(a.shape[1:3]) < 1`) and fail fast if violated.\n # Step 5: Branch on condition `input_range == 'uint8'`:\n # Step 6: Create evenly spaced index values in `rows`.\n raise NotImplementedError('Implement preprocess')", "solutionSnippet": "# Function `validate_splits` reference implementation:\ndef validate_splits(training,heldout):\n # Iterate over `data` to step through the computation:\n for data in (training,heldout):\n # Run `len` to compute `n`.\n n=len(data['images'])\n # Guard input contract (`np.asarray(data['labels']).shape != (n,) or len(data['ids']) != n or len(data['groups']) != n`) and fail fast if violated.\n if np.asarray(data['labels']).shape!=(n,) or len(data['ids'])!=n or len(data['groups'])!=n:raise ValueError('dataset shape contract')\n # Guard input contract (`len(set(data['ids'])) != n`) and fail fast if violated.\n if len(set(data['ids']))!=n:raise ValueError('duplicate image IDs')\n # Guard input contract (`not np.issubdtype(np.asarray(data['labels']).dtype, np.integer) or np.any((data['labels'] < 0) | (data['labels'] >= 3))`) and fail fast if violated.\n if not np.issubdtype(np.asarray(data['labels']).dtype,np.integer) or np.any((data['labels']<0)|(data['labels']>=3)):raise ValueError('class labels required')\n # Guard input contract (`set(training['ids']) & set(heldout['ids']) or set(training['groups']) & set(heldout['groups'])`) and fail fast if violated.\n if set(training['ids'])&set(heldout['ids']) or set(training['groups'])&set(heldout['groups']):raise ValueError('split leakage')\n # Construct dictionary `train_pixels` with the structured fields for this stage.\n train_pixels={digest_arrays(image) for image in training['images']}\n # Guard input contract (`train_pixels & {digest_arrays(image) for image in heldout['images']}`) and fail fast if violated.\n if train_pixels & {digest_arrays(image) for image in heldout['images']}:raise ValueError('duplicate image content across splits')\n\n# Function `preprocess` reference implementation:\ndef preprocess(images,layout='NHWC',input_range='uint8'):\n # Convert `a` to a host NumPy array for inspection or verification.\n a=np.asarray(images)\n # Guard input contract (`a.ndim != 4 or not len(a)`) and fail fast if violated.\n if a.ndim!=4 or not len(a):raise ValueError('nonempty rank-four image batch required')\n # Branch on condition `layout == 'NCHW'`:\n if layout=='NCHW':a=np.transpose(a,(0,2,3,1))\n elif layout!='NHWC':raise ValueError('unsupported layout')\n # Guard input contract (`a.shape[-1] not in (1, 3) or min(a.shape[1:3]) < 1`) and fail fast if violated.\n if a.shape[-1] not in (1,3) or min(a.shape[1:3])<1:raise ValueError('grayscale/RGB channels and positive size required')\n # Branch on condition `input_range == 'uint8'`:\n if input_range=='uint8':\n if a.dtype!=np.uint8:raise ValueError('uint8 input contract requires uint8 dtype')\n a=a.astype(np.float32)/255.\n elif input_range=='unit_float':\n if not np.issubdtype(a.dtype,np.floating) or not np.isfinite(a).all() or np.any((a<0)|(a>1)):raise ValueError('finite unit-range floating input required')\n a=a.astype(np.float32)\n else:raise ValueError('unsupported input range')\n # Initialize array `rows` with explicit values and shape.\n rows=(np.arange(8)*a.shape[1]//8)\n cols=(np.arange(8)*a.shape[2]//8)\n # Compute `a` as `a[:,rows][:,:,cols]`.\n a=a[:,rows][:,:,cols]\n # Branch on condition `a.shape[-1] == 3`:\n if a.shape[-1]==3:a=np.sum(a*np.array(PREPROCESS['rgb_weights'],np.float32),axis=-1,keepdims=True)\n # Return `np.ascontiguousarray(2 * a - 1, dtype=np.float32)` to the caller.\n return np.ascontiguousarray(2*a-1,dtype=np.float32)" }, { "id": "2", "title": "Transparent CNN baseline", "command": "python3 projects/image-harness/tests/check.py --stage 2 --implementation projects/image-harness/my_model.py", "evidence": "Shape trace, independent receptive field, parameter count and finite difference", "expected": "PASS stage 2", "diagnosis": "Check NHWC/HWIO axes and preserve singleton batches.", "codeGuide": "In `projects/image-harness/my_model.py`, implement `read_image(...)`, `load_external_manifest(...)` using `PIL`, `Image.open`, `contract`, `getattr`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 2 verifier.", "starterSnippet": "def read_image(path):\n 'External PNG/JPEG/etc through Pillow, explicit EXIF transpose and RGB conversion.'\n # Key APIs to use: `PIL`, `Image.open`, `contract`, `getattr`, `np.asarray`\n # Step 1: Import required JAX, NumPy, and standard-library modules.\n # Step 2: Enter managed runtime/context scope for this block:\n # Step 3: Inside block: Guard input contract (`image.width * image.height > 16000000`) and fail fast if violated.\n # Step 4: Inside block: Guard input contract (`getattr(image, 'n_frames', 1) != 1`) and fail fast if violated.\n raise NotImplementedError('Implement read_image')\n\ndef load_external_manifest(path):\n 'Load user-owned labeled images; relative paths stay inside manifest directory.\\n\\nJSON: split, source, license, items [{path,id,group,label}]. This loads supplied\\nfiles only and never downloads data. Split related subjects/sources before use.'\n # Key APIs to use: `disk`, `Path`, `json.loads`, `path.read_text`, `parent.resolve`\n # Step 1: Read or serialize artifact data on disk (`path`).\n # Step 2: Read or serialize artifact data on disk (`manifest`).\n # Step 3: Run `path.parent.resolve` to compute `root`.\n # Step 4: Guard input contract (`not all((manifest.get(k) for k in ('split', 'source', 'license', 'items')))`) and fail fast if violated.\n # Step 5: Evaluate `images` from the current inputs and state.\n # Step 6: Evaluate `labels` from the current inputs and state.\n raise NotImplementedError('Implement load_external_manifest')", "solutionSnippet": "# Function `read_image` reference implementation:\ndef read_image(path):\n \"\"\"External PNG/JPEG/etc through Pillow, explicit EXIF transpose and RGB conversion.\"\"\"\n # Import PIL (Image, ImageOps) for this computation.\n from PIL import Image,ImageOps\n # Enter `Image.open(path)` context block:\n with Image.open(path) as image:\n # Guard input contract (`image.width * image.height > 16000000`) and fail fast if violated.\n if image.width*image.height>16_000_000:raise ValueError('image exceeds teaching decoder pixel limit')\n # Guard input contract (`getattr(image, 'n_frames', 1) != 1`) and fail fast if violated.\n if getattr(image,'n_frames',1)!=1:raise ValueError('animated images are not supported')\n # Return `np.asarray(ImageOps.exif_transpose(image).convert('RGB'), dtype=np.uint8)[None]` to the caller.\n return np.asarray(ImageOps.exif_transpose(image).convert('RGB'),dtype=np.uint8)[None]\n\n# Function `load_external_manifest` reference implementation:\ndef load_external_manifest(path):\n \"\"\"Load user-owned labeled images; relative paths stay inside manifest directory.\n\n JSON: split, source, license, items [{path,id,group,label}]. This loads supplied\n files only and never downloads data. Split related subjects/sources before use.\n \"\"\"\n # Run `path.parent.resolve` to compute `root`.\n path=Path(path)\n manifest=json.loads(path.read_text())\n root=path.parent.resolve()\n # Guard input contract (`not all((manifest.get(k) for k in ('split', 'source', 'license', 'items')))`) and fail fast if violated.\n if not all(manifest.get(k) for k in ('split','source','license','items')):raise ValueError('data provenance and items required')\n # Initialize list `images` for the stage values.\n images=[]\n labels=[]\n ids=[]\n groups=[]\n file_hashes=[]\n # Loop over `item` in `manifest['items']`:\n for item in manifest['items']:\n # Evaluate the compound expression for `image_path`.\n image_path=(root/item['path']).resolve()\n # Guard input contract (`not image_path.is_relative_to(root)`) and fail fast if violated.\n if not image_path.is_relative_to(root):raise ValueError('image outside manifest directory')\n # Guard input contract (`item['label'] not in CLASSES`) and fail fast if violated.\n if item['label'] not in CLASSES:raise ValueError('unknown image label')\n # Guard input contract (`not item.get('id') or not item.get('group')`) and fail fast if violated.\n if not item.get('id') or not item.get('group'):raise ValueError('image and source-group IDs required')\n # Run `read_image` to compute `raw`.\n raw=read_image(image_path)\n # Create evenly spaced index values in `rows`.\n # Create evenly spaced index values in `cols`.\n rows=np.arange(8)*raw.shape[1]//8\n cols=np.arange(8)*raw.shape[2]//8\n # Append the current step result to `images`.\n # Append the current step result to `images`.\n images.append(raw[0][rows][:,cols])\n labels.append(CLASSES.index(item['label']))\n # Append the current step result to `ids`.\n # Append the current step result to `ids`.\n # Compute deterministic cryptographic digest `` for provenance verification.\n ids.append(item['id'])\n groups.append(item['group'])\n file_hashes.append(hashlib.sha256(image_path.read_bytes()).hexdigest())\n # Guard input contract (`len(set(ids)) != len(ids)`) and fail fast if violated.\n if len(set(ids))!=len(ids):raise ValueError('duplicate image IDs')\n # Return `dict(images=np.stack(images), labels=np.asarray(labels, np.int32), ids=ids, groups=groups, provenance=dict(kind='user-supplied image files', split=manifest['split'], source=manifest['source'], license=manifest['license'], file_hashes=file_hashes))` to the caller.\n return dict(images=np.stack(images),labels=np.asarray(labels,np.int32),ids=ids,groups=groups,\n provenance=dict(kind='user-supplied image files',split=manifest['split'],source=manifest['source'],license=manifest['license'],file_hashes=file_hashes))" }, { "id": "3", "title": "Actual augmented training", "command": "python3 projects/image-harness/tests/check.py --stage 3 --implementation projects/image-harness/my_model.py", "evidence": "Tiny-set fit, post-update train/held-out curves, seed replay", "expected": "PASS stage 3", "diagnosis": "Distinguish augmented minibatch pre-update loss from unaugmented full-set evaluation.", "codeGuide": "In `projects/image-harness/my_model.py`, implement `init_params(...)`, `forward(...)` using `random.split`, `random.PRNGKey`, `random.normal`, `jnp.zeros`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 3 verifier.", "starterSnippet": "def init_params(seed=0):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `random.split`, `random.PRNGKey`, `random.normal`, `jnp.zeros`\n # Step 1: Initialize explicit deterministic PRNG key `(a, b)`.\n # Step 2: Return `dict(conv=jax.random.normal(a, (3, 3, 1, 6)) * 0.3, conv_bias=jnp.zeros(6), head=jax.random.normal(b, (6, 3)) * 0.2, head_bias=jnp.zeros(3))` to the caller.\n raise NotImplementedError('Implement init_params')\n\ndef forward(params,x,compute='float32',return_hidden=False):\n 'Valid 3x3 convolution, ReLU, spatial mean and class logits.'\n # Key APIs to use: `lax.conv_general_dilated`, `jnp.asarray`, `astype`, `nn.relu`, `hidden.mean`\n # Step 1: Cast or evaluate `dtype` in explicit floating-point precision.\n # Step 2: Create device-backed JAX array `conv`.\n # Step 3: Run `jax.nn.relu` to compute `hidden`.\n # Step 4: Reduce across the target axis to summarize `pooled`.\n # Step 5: Cast or evaluate `logits` in explicit floating-point precision.\n # Step 6: Return `(logits, pooled) if return_hidden else logits` to the caller.\n raise NotImplementedError('Implement forward')", "solutionSnippet": "# Function `init_params` reference implementation:\ndef init_params(seed=0):\n # Initialize explicit deterministic PRNG key `(a, b)`.\n a,b=jax.random.split(jax.random.PRNGKey(seed))\n # Return `dict(conv=jax.random.normal(a, (3, 3, 1, 6)) * 0.3, conv_bias=jnp.zeros(6), head=jax.random.normal(b, (6, 3)) * 0.2, head_bias=jnp.zeros(3))` to the caller.\n return dict(conv=jax.random.normal(a,(3,3,1,6))*.3,conv_bias=jnp.zeros(6),\n head=jax.random.normal(b,(6,3))*.2,head_bias=jnp.zeros(3))\n\n# Function `forward` reference implementation:\ndef forward(params,x,compute='float32',return_hidden=False):\n # Cast or evaluate `dtype` in explicit floating-point precision.\n dtype={'float32':jnp.float32,'float16':jnp.float16,'bfloat16':jnp.bfloat16}[compute]\n # Create device-backed JAX array `conv`.\n conv=jax.lax.conv_general_dilated(jnp.asarray(x,dtype),params['conv'].astype(dtype),(1,1),'VALID',dimension_numbers=('NHWC','HWIO','NHWC'),preferred_element_type=jnp.float32)\n # Run `jax.nn.relu` to compute `hidden`.\n hidden=jax.nn.relu(conv+params['conv_bias'])\n # Reduce across the target axis to summarize `pooled`.\n pooled=hidden.mean(axis=(1,2))\n # Cast or evaluate `logits` in explicit floating-point precision.\n logits=jax.lax.dot_general(pooled.astype(dtype),params['head'].astype(dtype),(((1,),(0,)),((),())),preferred_element_type=jnp.float32)+params['head_bias']\n # Return `(logits, pooled) if return_hidden else logits` to the caller.\n return (logits,pooled) if return_hidden else logits" }, { "id": "4", "title": "Full-state recovery", "command": "python3 projects/image-harness/tests/check.py --stage 4 --implementation projects/image-harness/my_model.py", "evidence": "Fresh-process next IDs, augmented inputs, losses and optimizer/key/cursor equality", "expected": "PASS stage 4", "diagnosis": "Weights alone omit momentum, randomness and the next sampler position.", "codeGuide": "In `projects/image-harness/my_model.py`, implement `objective(...)`, `start(...)` using `jnp.asarray`, `contract`, `or`, `jnp.issubdtype`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 4 verifier.", "starterSnippet": "def objective(params,x,labels):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `jnp.asarray`, `contract`, `or`, `jnp.issubdtype`, `forward`\n # Step 1: Create device-backed JAX array `labels`.\n # Step 2: Guard input contract (`labels.ndim != 1 or labels.shape[0] != x.shape[0] or (not jnp.issubdtype(labels.dtype, jnp.integer))`) and fail fast if violated.\n # Step 3: Run `forward` to compute `logits`.\n # Step 4: Return `-jnp.mean(jnp.take_along_axis(jax.nn.log_softmax(logits), labels[:, None], axis=1))` to the caller.\n raise NotImplementedError('Implement objective')\n\ndef start(data,seed=0,settings=None):\n 'Initialize parameters, momentum, key, permutation and data cursor.'\n # Key APIs to use: `config`, `init_params`, `random.split`, `random.PRNGKey`, `np.asarray`\n # Step 1: Run `config` to compute `settings`.\n # Step 2: Run `init_params` to compute `params`.\n # Step 3: Initialize explicit deterministic PRNG key `(key, order_key)`.\n # Step 4: Convert `order` to a host NumPy array for inspection or verification.\n # Step 5: Return `dict(params=params, velocity=jax.tree.map(jnp.zeros_like, params), key=key, order=order, position=0, epoch=0, step=0, seed=seed, data_hash=dataset_hash(data), settings=settings)` to the caller.\n raise NotImplementedError('Implement start')", "solutionSnippet": "# Function `objective` reference implementation:\ndef objective(params,x,labels):\n # Create device-backed JAX array `labels`.\n labels=jnp.asarray(labels)\n # Guard input contract (`labels.ndim != 1 or labels.shape[0] != x.shape[0] or (not jnp.issubdtype(labels.dtype, jnp.integer))`) and fail fast if violated.\n if labels.ndim!=1 or labels.shape[0]!=x.shape[0] or not jnp.issubdtype(labels.dtype,jnp.integer):raise ValueError('one integer label per image required')\n # Run `forward` to compute `logits`.\n logits=forward(params,x)\n # Return `-jnp.mean(jnp.take_along_axis(jax.nn.log_softmax(logits), labels[:, None], axis=1))` to the caller.\n return -jnp.mean(jnp.take_along_axis(jax.nn.log_softmax(logits),labels[:,None],axis=1))\n\n# Function `start` reference implementation:\ndef start(data,seed=0,settings=None):\n # Run `config` to compute `settings`.\n # Run `init_params` to compute `params`.\n settings=config(**(settings or {}))\n params=init_params(seed)\n # Initialize explicit deterministic PRNG key `(key, order_key)`.\n key,order_key=jax.random.split(jax.random.PRNGKey(seed+100))\n # Convert `order` to a host NumPy array for inspection or verification.\n order=np.asarray(jax.random.permutation(order_key,len(data['images'])))\n # Return `dict(params=params, velocity=jax.tree.map(jnp.zeros_like, params), key=key, order=order, position=0, epoch=0, step=0, seed=seed, data_hash=dataset_hash(data), settings=settings)` to the caller.\n return dict(params=params,velocity=jax.tree.map(jnp.zeros_like,params),key=key,order=order,\n position=0,epoch=0,step=0,seed=seed,data_hash=dataset_hash(data),settings=settings)" }, { "id": "5", "title": "Frozen evaluation and errors", "command": "python3 projects/image-harness/tests/check.py --stage 5 --implementation projects/image-harness/my_model.py", "evidence": "Unequal-batch metric parity, confusion counts and aligned corruption errors", "expected": "PASS stage 5", "diagnosis": "Read rows as truth, columns as predictions; inspect removed information before explaining failure.", "codeGuide": "In `projects/image-harness/my_model.py`, implement `_update(...)`, `advance(...)` using `subkeys`, `random.split`, `random.bernoulli`, `jnp.where`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 5 verifier.", "starterSnippet": "def _update(params,velocity,key,x,y,learning_rate,momentum):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `subkeys`, `random.split`, `random.bernoulli`, `jnp.where`, `jnp.clip`\n # Step 1: Split the PRNG key deterministically into independent subkeys (`(key, noise_key, flip_key)`).\n # Step 2: Draw pseudorandom samples for `flips` using the explicit RNG state.\n # Step 3: Combine or mask array elements to form `augmented`.\n # Step 4: Draw pseudorandom samples for `augmented` using the explicit RNG state.\n # Step 5: Evaluate both scalar loss and parameter gradients in one pass (`(loss, grad)`).\n # Step 6: Apply leaf-wise transformation across the PyTree to produce `velocity`.\n raise NotImplementedError('Implement _update')\n\ndef advance(state,data,updates=1):\n 'Complete actual augmented minibatch updates and preserve next-input state.'\n # Key APIs to use: `contract`, `dataset_hash`, `preprocess`, `random.split`, `np.asarray`\n # Step 1: Guard input contract (`dataset_hash(data) != state['data_hash']`) and fail fast if violated.\n # Step 2: Guard input contract (`not isinstance(updates, int) or updates < 1`) and fail fast if violated.\n # Step 3: Evaluate `state` and convert the result into Python scalar/collection `s`.\n # Step 4: Evaluate `records` from the current inputs and state.\n # Step 5: Run `len` to compute `n`.\n # Step 6: Evaluate `batch` from the current inputs and state.\n raise NotImplementedError('Implement advance')", "solutionSnippet": "# Function `_update` reference implementation:\ndef _update(params,velocity,key,x,y,learning_rate,momentum):\n # Split the PRNG key deterministically into independent subkeys (`(key, noise_key, flip_key)`).\n key,noise_key,flip_key=jax.random.split(key,3)\n # Draw pseudorandom samples for `flips` using the explicit RNG state.\n flips=jax.random.bernoulli(flip_key,.5,(len(x),1,1,1))\n # Combine or mask array elements to form `augmented`.\n augmented=jnp.where(flips,x[:,:,::-1,:],x)\n # Draw pseudorandom samples for `augmented` using the explicit RNG state.\n augmented=jnp.clip(augmented+.035*jax.random.normal(noise_key,x.shape),-1,1)\n # Evaluate both scalar loss and parameter gradients in one pass (`(loss, grad)`).\n loss,grad=jax.value_and_grad(objective)(params,augmented,y)\n # Apply leaf-wise transformation across the PyTree to produce `velocity`.\n velocity=jax.tree.map(lambda v,g:momentum*v+g,velocity,grad)\n # Apply leaf-wise transformation across the PyTree to produce `params`.\n params=jax.tree.map(lambda p,v:p-learning_rate*v,params,velocity)\n # Return `(params, velocity, key, loss, augmented)` to the caller.\n return params,velocity,key,loss,augmented\n\n# Function `advance` reference implementation:\ndef advance(state,data,updates=1):\n # Guard input contract (`dataset_hash(data) != state['data_hash']`) and fail fast if violated.\n if dataset_hash(data)!=state['data_hash']:raise ValueError('dataset changed since checkpoint')\n # Guard input contract (`not isinstance(updates, int) or updates < 1`) and fail fast if violated.\n if not isinstance(updates,int) or updates<1:raise ValueError('positive update count required')\n # Evaluate `state` and convert the result into Python scalar/collection `s`.\n # Run `len` to compute `n`.\n # Compute `s` as `dict(state)`.\n s=dict(state)\n records=[]\n n=len(data['images'])\n batch=s['settings']['batch_size']\n # Guard input contract (`batch > n`) and fail fast if violated.\n if batch>n:raise ValueError('batch exceeds dataset')\n # Run `preprocess` to compute `all_x`.\n all_x=preprocess(data['images'])\n # Repeat the update loop over `range(updates)` steps:\n for _ in range(updates):\n # Branch on condition `s['position'] + batch > n`:\n if s['position']+batch>n:\n s['key'],k=jax.random.split(s['key'])\n s['order']=np.asarray(jax.random.permutation(k,n))\n s['position']=0\n s['epoch']+=1\n # Compute `indices` as `s['order'][s['position']:s['position']+batch]`.\n indices=s['order'][s['position']:s['position']+batch]\n # Create device-backed JAX array `(p, v, k, loss, aug)`.\n p,v,k,loss,aug=_update(s['params'],s['velocity'],s['key'],jnp.asarray(all_x[indices]),jnp.asarray(data['labels'][indices]),s['settings']['learning_rate'],s['settings']['momentum'])\n # Synchronize host execution until asynchronous device computation completes.\n # Synchronize host execution until asynchronous device computation completes.\n jax.block_until_ready((p,v,k,loss,aug))\n s.update(params=p,velocity=v,key=k,position=s['position']+batch,step=s['step']+1)\n # Append the current step result to `records`.\n records.append(dict(step=s['step'],loss=float(loss),ids=[data['ids'][i] for i in indices],augmented_hash=digest_arrays(aug)))\n # Return `(s, records)` to the caller.\n return s,records" }, { "id": "6", "title": "Calibrated precision", "command": "python3 projects/image-harness/tests/check.py --stage 6 --implementation projects/image-harness/my_model.py", "evidence": "Training-only scales, actual integer accumulations, clipping, logit errors and byte counts", "expected": "PASS stage 6", "diagnosis": "Separate weight, activation, accumulation and float intermediate operations.", "codeGuide": "In `projects/image-harness/my_model.py`, implement `evaluate(...)`, `metrics(...)` using `contract`, `preprocess`, `np.asarray`, `np.issubdtype`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 6 verifier.", "starterSnippet": "def evaluate(params,data,batch_size=17,compute='float32'):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `contract`, `preprocess`, `np.asarray`, `np.issubdtype`, `np.any`\n # Step 1: Guard input contract (`batch_size < 1 or not len(data['images'])`) and fail fast if violated.\n # Step 2: Run `preprocess` to compute `x`.\n # Step 3: Convert `y` to a host NumPy array for inspection or verification.\n # Step 4: Guard input contract (`y.shape != (len(x),) or not np.issubdtype(y.dtype, np.integer) or np.any((y < 0) | (y >= 3))`) and fail fast if violated.\n # Step 5: Create device-backed JAX array `logits`.\n # Step 6: Return `metrics(logits, y, data['ids'])` to the caller.\n raise NotImplementedError('Implement evaluate')\n\ndef metrics(logits,labels,ids):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `np.asarray`, `contract`, `or`, `np.isfinite`, `all`\n # Step 1: Convert `logits` to a host NumPy array for inspection or verification.\n # Step 2: Convert `labels` to a host NumPy array for inspection or verification.\n # Step 3: Guard input contract (`logits.ndim != 2 or logits.shape[1] != 3 or len(logits) == 0 or (not np.isfinite(logits).all()) or (labels.shape != (len(logits),)) or (not np.issubdtype(labels.dtype, np.integer)) or np.any((labels < 0) | (labels >= 3)) or (len(ids) != len(logits))`) and fail fast if violated.\n # Step 4: Reduce across the target axis to summarize `shifted`.\n # Step 5: Create evenly spaced index values in `losses`.\n # Step 6: Run `np.argmax` to compute `predicted`.\n raise NotImplementedError('Implement metrics')", "solutionSnippet": "# Function `evaluate` reference implementation:\ndef evaluate(params,data,batch_size=17,compute='float32'):\n # Guard input contract (`batch_size < 1 or not len(data['images'])`) and fail fast if violated.\n if batch_size<1 or not len(data['images']):raise ValueError('positive evaluation batch and nonempty data required')\n # Run `preprocess` to compute `x`.\n # Convert `y` to a host NumPy array for inspection or verification.\n x=preprocess(data['images'])\n y=np.asarray(data['labels'])\n # Guard input contract (`y.shape != (len(x),) or not np.issubdtype(y.dtype, np.integer) or np.any((y < 0) | (y >= 3))`) and fail fast if violated.\n if y.shape!=(len(x),) or not np.issubdtype(y.dtype,np.integer) or np.any((y<0)|(y>=3)):raise ValueError('valid integer vector labels required')\n # Create device-backed JAX array `logits`.\n logits=np.concatenate([np.asarray(forward(params,jnp.asarray(x[i:i+batch_size]),compute)) for i in range(0,len(x),batch_size)])\n # Return `metrics(logits, y, data['ids'])` to the caller.\n return metrics(logits,y,data['ids'])\n\n# Function `metrics` reference implementation:\ndef metrics(logits,labels,ids):\n # Convert `logits` to a host NumPy array for inspection or verification.\n # Convert `labels` to a host NumPy array for inspection or verification.\n logits=np.asarray(logits,dtype=np.float64)\n labels=np.asarray(labels)\n # Guard input contract (`logits.ndim != 2 or logits.shape[1] != 3 or len(logits) == 0 or (not np.isfinite(logits).all()) or (labels.shape != (len(logits),)) or (not np.issubdtype(labels.dtype, np.integer)) or np.any((labels < 0) | (labels >= 3)) or (len(ids) != len(logits))`) and fail fast if violated.\n if logits.ndim!=2 or logits.shape[1]!=3 or len(logits)==0 or not np.isfinite(logits).all() or labels.shape!=(len(logits),) or not np.issubdtype(labels.dtype,np.integer) or np.any((labels<0)|(labels>=3)) or len(ids)!=len(logits):raise ValueError('invalid evaluation arrays')\n # Reduce across the target axis to summarize `shifted`.\n shifted=logits-logits.max(1,keepdims=True)\n # Create evenly spaced index values in `losses`.\n losses=np.log(np.exp(shifted).sum(1))-shifted[np.arange(len(labels)),labels]\n # Run `np.argmax` to compute `predicted`.\n # Allocate initialized array `confusion` with the specified shape and dtype.\n predicted=np.argmax(logits,1)\n confusion=np.zeros((3,3),int)\n # Run `np.add.at` to perform the next check or state transition.\n np.add.at(confusion,(labels,predicted),1)\n # Return `dict(count=len(labels), loss=float(losses.mean()), loss_sum=float(losses.sum()), correct=int((predicted == labels).sum()), accuracy=float((predicted == labels).mean()), confusion=confusion.tolist(), error_ids=[ids[i] for i in np.flatnonzero(predicted != labels)], logits=logits, predictions=predicted)` to the caller.\n return dict(count=len(labels),loss=float(losses.mean()),loss_sum=float(losses.sum()),correct=int((predicted==labels).sum()),\n accuracy=float((predicted==labels).mean()),confusion=confusion.tolist(),error_ids=[ids[i] for i in np.flatnonzero(predicted!=labels)],logits=logits,predictions=predicted)" }, { "id": "7", "title": "Serialized inference artifact", "command": "python3 projects/image-harness/tests/check.py --stage 7 --implementation projects/image-harness/my_model.py", "evidence": "Actual JAX graph reload in a fresh interpreter, signatures and contract guards", "expected": "PASS stage 7", "diagnosis": "Verify payload hashes and preprocessing/class mappings before inference.", "codeGuide": "In `projects/image-harness/my_model.py`, implement `save_checkpoint(...)`, `restore_checkpoint(...)` using `disk`, `Path`, `parent.mkdir`, `np.asarray`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 7 verifier.", "starterSnippet": "def save_checkpoint(path,state):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `disk`, `Path`, `parent.mkdir`, `np.asarray`, `items`\n # Step 1: Read or serialize artifact data on disk (`path`).\n # Step 2: Execute the next step of the computation.\n # Step 3: Convert `arrays` to a host NumPy array for inspection or verification.\n # Step 4: Evaluate `meta` from the current inputs and state.\n # Step 5: Execute the next step of the computation.\n # Step 6: Run `digest_json` to compute `meta['metadata_hash']`.\n raise NotImplementedError('Implement save_checkpoint')\n\ndef restore_checkpoint(path,data,settings=None):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `np.load`, `disk`, `copy`, `json.loads`, `meta.pop`\n # Step 1: Enter managed runtime/context scope for this block:\n # Step 2: Inside block: Evaluate `arrays` from the current inputs and state.\n # Step 3: Inside block: Read or serialize artifact data on disk (`meta`).\n # Step 4: Run `meta.pop` to compute `metadata_hash`.\n # Step 5: Guard input contract (`metadata_hash != digest_json(meta)`) and fail fast if violated.\n # Step 6: Guard input contract (`meta['jax_version'] != jax.__version__ or meta['numpy_version'] != np.__version__`) and fail fast if violated.\n raise NotImplementedError('Implement restore_checkpoint')", "solutionSnippet": "# Function `save_checkpoint` reference implementation:\ndef save_checkpoint(path,state):\n # Compute `path` as `Path(path)`.\n path=Path(path)\n path.parent.mkdir(parents=True,exist_ok=True)\n # Convert `arrays` to a host NumPy array for inspection or verification.\n arrays={**{'p_'+k:np.asarray(v) for k,v in state['params'].items()},**{'v_'+k:np.asarray(v) for k,v in state['velocity'].items()},'key':np.asarray(state['key']),'order':np.asarray(state['order'])}\n # Construct dictionary `meta` with the structured fields for this stage.\n meta={k:state[k] for k in ('step','position','epoch','seed','data_hash','settings')}\n meta.update(version=1,jax_version=jax.__version__,numpy_version=np.__version__,preprocess=PREPROCESS,arrays_hash=digest_arrays(*[arrays[k] for k in sorted(arrays)]))\n # Run `digest_json` to compute `meta['metadata_hash']`.\n meta['metadata_hash']=digest_json(meta)\n # Run `tempfile.mkstemp` to compute `(fd, name)`.\n fd,name=tempfile.mkstemp(dir=path.parent,suffix='.npz')\n try:\n with os.fdopen(fd,'wb') as out:\n np.savez(out,**arrays,metadata=np.array(json.dumps(meta,sort_keys=True)))\n out.flush()\n os.fsync(out.fileno())\n os.replace(name,path)\n finally:\n if os.path.exists(name):os.unlink(name)\n\n# Function `restore_checkpoint` reference implementation:\ndef restore_checkpoint(path,data,settings=None):\n # Enter managed runtime/context scope for this block:\n with np.load(path,allow_pickle=False) as bundle:\n # Read or serialize artifact data on disk (`meta`).\n arrays={k:bundle[k].copy() for k in bundle.files if k!='metadata'}\n meta=json.loads(str(bundle['metadata']))\n # Run `meta.pop` to compute `metadata_hash`.\n metadata_hash=meta.pop('metadata_hash',None)\n # Guard input contract (`metadata_hash != digest_json(meta)`) and fail fast if violated.\n if metadata_hash!=digest_json(meta):raise ValueError('checkpoint metadata integrity mismatch')\n # Guard input contract (`meta['jax_version'] != jax.__version__ or meta['numpy_version'] != np.__version__`) and fail fast if violated.\n if meta['jax_version']!=jax.__version__ or meta['numpy_version']!=np.__version__:raise ValueError('checkpoint runtime version mismatch')\n # Guard input contract (`meta['version'] != 1 or meta['preprocess'] != PREPROCESS or meta['data_hash'] != dataset_hash(data) or (meta['settings'] != config(**settings or {}))`) and fail fast if violated.\n if meta['version']!=1 or meta['preprocess']!=PREPROCESS or meta['data_hash']!=dataset_hash(data) or meta['settings']!=config(**(settings or {})):raise ValueError('checkpoint data/preprocess/config mismatch')\n # Guard input contract (`meta['arrays_hash'] != digest_arrays(*[arrays[k] for k in sorted(arrays)])`) and fail fast if violated.\n if meta['arrays_hash']!=digest_arrays(*[arrays[k] for k in sorted(arrays)]):raise ValueError('checkpoint integrity mismatch')\n # Guard input contract (`arrays['key'].shape != (2,) or arrays['key'].dtype != np.uint32`) and fail fast if violated.\n if arrays['key'].shape!=(2,) or arrays['key'].dtype!=np.uint32:raise ValueError('invalid random key')\n # Guard input contract (`sorted(arrays['order'].tolist()) != list(range(len(data['images']))) or not 0 <= meta['position'] <= len(data['images']) or meta['step'] < 0 or (meta['epoch'] < 0)`) and fail fast if violated.\n if sorted(arrays['order'].tolist())!=list(range(len(data['images']))) or not 0<=meta['position']<=len(data['images']) or meta['step']<0 or meta['epoch']<0:raise ValueError('invalid sampler state')\n # Create device-backed JAX array `params`.\n # Create device-backed JAX array `velocity`.\n params={k: jnp.asarray(arrays['p_'+k]) for k in init_params()}\n velocity={k:jnp.asarray(arrays['v_'+k]) for k in params}\n # Loop over `(k, template)` in `init_params().items()`:\n for k,template in init_params().items():\n # Guard input contract (`params[k].shape != template.shape or velocity[k].shape != template.shape or (not np.isfinite(params[k]).all()) or (not np.isfinite(velocity[k]).all())`) and fail fast if violated.\n if params[k].shape!=template.shape or velocity[k].shape!=template.shape or not np.isfinite(params[k]).all() or not np.isfinite(velocity[k]).all():raise ValueError('invalid parameter or optimizer state')\n # Return `dict(params=params, velocity=velocity, key=jnp.asarray(arrays['key']), order=arrays['order'], **{k: meta[k] for k in ('step', 'position', 'epoch', 'seed', 'data_hash', 'settings')})` to the caller.\n return dict(params=params,velocity=velocity,key=jnp.asarray(arrays['key']),order=arrays['order'],**{k:meta[k] for k in ('step','position','epoch','seed','data_hash','settings')})" }, { "id": "8", "title": "Measured serving adapter", "command": "python3 projects/image-harness/tests/check.py --stage 8 --implementation projects/image-harness/my_model.py", "evidence": "Warmup, repeated completed CPU inference, failed candidate and rollback trace", "expected": "PASS stage 8", "diagnosis": "State decoding exclusions, actual device and timing units; avoid unsupported target speed claims.", "codeGuide": "In `projects/image-harness/my_model.py`, implement `patches(...)`, `quantize(...)`, `calibrate(...)`, `integer_forward(...)`, `export_model(...)`, `load_model(...)`, `infer(...)`, `benchmark(...)`, `select_runtime(...)` using `np.stack`, `reshape`, `np.clip`, `np.rint`. Follow the numbered `# Step 1..N` comments in the starter scaffold below before running the Stage 8 verifier.", "starterSnippet": "def patches(x):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `np.stack`, `reshape`\n # Step 1: Return `np.stack([x[:, i:i + 6, j:j + 6, :] for i in range(3) for j in range(3)], axis=-2).reshape(len(x), 6, 6, 9)` to the caller.\n raise NotImplementedError('Implement patches')\n\ndef quantize(values,scale):\n 'Implement the contract described in README.md and the stage checks.'\n # Key APIs to use: `np.clip`, `np.rint`, `np.asarray`, `astype`\n # Step 1: Return `np.clip(np.rint(np.asarray(values) / scale), -127, 127).astype(np.int8)` to the caller.\n raise NotImplementedError('Implement quantize')\n\ndef calibrate(params,data,percentile=99.):\n 'Fit per-output-channel weight scales and input/pooled activation scales on training data.'\n # Key APIs to use: `contract`, `preprocess`, `forward`, `jnp.asarray`, `max`\n # Step 1: Guard input contract (`not 0 < percentile <= 100`) and fail fast if violated.\n # Step 2: Run `preprocess` to compute `x`.\n # Step 3: Create device-backed JAX array `(_, hidden)`.\n # Step 4: Run `max` to compute `input_scale`.\n # Step 5: Run `max` to compute `hidden_scale`.\n # Step 6: Convert `conv` to a host NumPy array for inspection or verification.\n raise NotImplementedError('Implement calibrate')\n\ndef integer_forward(quantized,x,accumulator='int32'):\n 'Perform int8 products with int32 accumulation; report activation clipping.'\n # Key APIs to use: `contract`, `np.asarray`, `quantize`, `patches`, `astype`\n # Step 1: Guard input contract (`accumulator != 'int32'`) and fail fast if violated.\n # Step 2: Evaluate `q` from the current inputs and state.\n # Step 3: Convert `raw` to a host NumPy array for inspection or verification.\n # Step 4: Run `quantize` to compute `ix`.\n # Step 5: Perform matrix contraction / projection to compute `acc`.\n # Step 6: Cast or evaluate `hidden` in explicit floating-point precision.\n raise NotImplementedError('Implement integer_forward')\n\ndef export_model(folder,state,quantized):\n 'Serialize actual JAX graphs for batch sizes 1 and 4 plus integer data and preprocessing contract.'\n # Key APIs to use: `disk`, `Path`, `contract`, `folder.exists`, `folder.mkdir`\n # Step 1: Read or serialize artifact data on disk (`folder`).\n # Step 2: Guard input contract (`folder.exists()`) and fail fast if violated.\n # Step 3: Run `folder.mkdir` to perform the next check or state transition.\n # Step 4: Loop over `batch` in `(1, 4)`:\n # Step 5: Inside block: Cast or evaluate `artifact` in explicit floating-point precision.\n # Step 6: Inside block: Execute the next step of the computation.\n raise NotImplementedError('Implement export_model')\n\ndef load_model(folder,expected_preprocess=PREPROCESS):\n 'Verify all hashes/contracts and deserialize JAX computations.'\n # Key APIs to use: `disk`, `Path`, `json.loads`, `read_text`, `contract`\n # Step 1: Read or serialize artifact data on disk (`folder`).\n # Step 2: Read or serialize artifact data on disk (`meta`).\n # Step 3: Guard input contract (`meta['version'] != 1 or meta['preprocess'] != expected_preprocess or meta['classes'] != CLASSES or (meta['batches'] != [1, 4]) or (meta['integer_accumulator'] != 'int32')`) and fail fast if violated.\n # Step 4: Evaluate `names` from the current inputs and state.\n # Step 5: Guard input contract (`set(meta['files']) != set(names)`) and fail fast if violated.\n # Step 6: Loop over `name` in `names`:\n raise NotImplementedError('Implement load_model')\n\ndef infer(runtime,images,layout='NHWC',input_range='uint8',precision='float32'):\n 'Validate/preprocess raw images and call the selected runtime.'\n # Key APIs to use: `preprocess`, `integer_forward`, `contract`, `call`, `jnp.asarray`\n # Step 1: Run `preprocess` to compute `x`.\n # Step 2: Branch on condition `precision == 'int8'`:\n # Step 3: Guard input contract (`precision != 'float32'`) and fail fast if violated.\n # Step 4: Evaluate `outputs` from the current inputs and state.\n # Step 5: Evaluate `position` from the current inputs and state.\n # Step 6: Return `np.concatenate(outputs)` to the caller.\n raise NotImplementedError('Implement infer')\n\ndef benchmark(runtime,images,repeats=12,precision='float32'):\n 'Warm up and repeatedly synchronize measured inference.'\n # Key APIs to use: `contract`, `time.perf_counter`, `infer`, `samples.append`, `jax.default_backend`\n # Step 1: Guard input contract (`repeats < 3`) and fail fast if violated.\n # Step 2: Record execution timing or profiler trace in `start`.\n # Step 3: Execute the next step of the computation.\n # Step 4: Record execution timing or profiler trace in `warmup`.\n # Step 5: Evaluate `samples` from the current inputs and state.\n # Step 6: Repeat the update loop over `range(repeats)` steps:\n raise NotImplementedError('Implement benchmark')\n\ndef select_runtime(candidate_folder,canary_images,expected_logits):\n 'Validate a candidate before the caller swaps its active runtime reference.'\n # Key APIs to use: `load_model`, `infer`, `contract`, `np.shape`, `np.allclose`\n # Step 1: Run `load_model` to compute `candidate`.\n # Step 2: Run `infer` to compute `actual`.\n # Step 3: Guard input contract (`np.shape(actual) != np.shape(expected_logits) or not np.allclose(actual, expected_logits, rtol=2e-05, atol=2e-05)`) and fail fast if violated.\n # Step 4: Return `candidate` to the caller.\n raise NotImplementedError('Implement select_runtime')", "solutionSnippet": "# Function `patches` reference implementation:\ndef patches(x):\n # Return `np.stack([x[:, i:i + 6, j:j + 6, :] for i in range(3) for j in range(3)], axis=-2).reshape(len(x), 6, 6, 9)` to the caller.\n return np.stack([x[:,i:i+6,j:j+6,:] for i in range(3) for j in range(3)],axis=-2).reshape(len(x),6,6,9)\n\n# Function `quantize` reference implementation:\ndef quantize(values,scale):\n # Return `np.clip(np.rint(np.asarray(values) / scale), -127, 127).astype(np.int8)` to the caller.\n return np.clip(np.rint(np.asarray(values)/scale),-127,127).astype(np.int8)\n\n# Function `calibrate` reference implementation:\ndef calibrate(params,data,percentile=99.):\n # Guard input contract (`not 0 < percentile <= 100`) and fail fast if violated.\n if not 0