Systems Specialization · CPU ReferenceCPU reference; target qualification separateLesson 7 of 8 in Phase 16

Phase 16 / Workload operations

LLMOps: version prompts, evaluate behavior and trace requests

Revision v0.7.0 · 2026-10-20

Lesson · about 75 minutes · CPU practice

Evaluate a versioned text application with a complete case inventory, slice-level answer contracts and inspectable MLflow traces.

Phase study guide: preparation, milestones and project evidence

Before you start

Before this: MLOps: data contracts, CI gates and monitoring
Hardware: CPU

Download notebook

Other downloads

Python worked examples Lesson text

Setup help: Python, notebook, or Cloud TPU VM

Use the tested CPU packages (the recorded environment uses Python 3.14.3). Other Python versions and accelerator environments require separate checks. The optional Cloud TPU specialization explains the target-specific execution boundaries. Logical CPU devices practice partitioning; they do not simulate TPU performance.

Download the repository ZIP or use your existing checkout. Extract it and open a terminal in the folder containing requirements-cpu.txt. The beginner ZIP is separate; these lessons use the full repository folder.

macOS / Linux terminal · repository folder

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-cpu.txt

PowerShell · repository folder

py -3 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-cpu.txt

After activation, run the complete worked example from that same folder:

macOS / Linux terminal · activated environment

python phases/16-operations/07-llmops-prompts-evaluation-and-traces/code/main.py

PowerShell command · repository folder

.\.venv\Scripts\python.exe phases/16-operations/07-llmops-prompts-evaluation-and-traces/code/main.py

For an incremental build, create main.py in your own practice folder, append each build block in order, and run python main.py with this environment active. Keep that file separate from the downloaded reference solution.

Download this lesson’s notebook. Open it with a Python kernel using the course environment. Run cells from the top after restarting the kernel; experiments depend on earlier definitions.

For local Jupyter, first install the course requirements using the Local Python steps, then use your environment’s Python to start Jupyter:

macOS / Linux terminal · activated course environment

python -m pip install jupyterlab
python -m jupyter lab

PowerShell command · repository folder

.\.venv\Scripts\python.exe -m pip install jupyterlab
.\.venv\Scripts\python.exe -m jupyter lab

Select the downloaded operations-07.ipynb file in JupyterLab. The notebook contains reference solutions; make your prediction and attempt each practice task before opening them. Restart the kernel before changing logical device setup.

Check: Observed replay scores: [{'overall': 0.9, 'answerable': 1.0, 'unanswerable': 0.0}, {'overall': 1.0, 'answerable': 1.0, 'unanswerable': 1.0}] Two real MLflow traces; no LLM API was called and no general model quality was evaluated.

Revisit the guided CPU setup · Open 19: TPU Setup. Package instructions are CPU instructions; Windows and accelerator environments need their own validation.

Phase 16.07 Stage 1 of 5 · Understand

1 Understand 2 Build & Run 3 Experiment 4 Exercise & Check 5 Evidence

What you will be able to do

Version the application boundary
Define cases before changing the prompt
Read the rare-case failure
Trace enough to diagnose, without retaining everything

The problem

A prompt edit improves ordinary answers but makes the system confidently answer a question its documents cannot support. Let’s build a small regression suite that exposes the failure and save real MLflow traces without calling a paid model.

The idea

An LLM application combines prompts, retrieved context, model configuration and postprocessing. Version those components and keep a trace that explains a failed case. A model name alone does not identify the behavior of the whole application.

Trace a response through the system that produced it

If a response is unsupported, first ask whether relevant evidence reached the model, whether the prompt requested grounded behavior and whether the evaluation checks the intended claim. Changing the model cannot necessarily repair a missing retrieval candidate.

Link case IDs to prompt, retriever, model and evaluator versions. Re-run a fixed case set after a change and inspect both repaired cases and regressions. An average pass rate can conceal a newly broken critical case.

The lesson uses a bounded fixture to teach release logic. Preserve that scope: passing a mocked or non-LLM check is separate from actual model behavior. Qualify the full application with executed traces when replacing the fixture.

\text{Digest}(\text{Image}) = \operatorname{sha256}(\text{Layers} \;\|\; \text{Lockfile}), \qquad P(\text{reproduce}) = 1

Version the application boundary

For a text generator, record the model revision, tokenizer, prompt template and decoding settings. For retrieval, add corpus/index versions, chunking and retrieval configuration. For tools, version schemas and implementations and define permission boundaries. MLflow’s prompt registry can store versions; an alias is convenient for development, while an evaluated release should resolve an immutable prompt version. The connected lab actually registers and reloads a prompt version.

Define cases before changing the prompt

The companion replays controlled answer objects against ten public fixtures: nine answerable cases and one unsupported question. An answer contains text, citation IDs and an abstention flag. The evaluator checks the output schema, exact expected answer for these deliberately simple fixtures, allowed citation IDs and abstention for the unsupported case. A citation ID being allowed does not generally prove that a claim is entailed by a document; real tasks need claim-level checks and human calibration.

Read the rare-case failure

The always-answer candidate passes all nine ordinary cases and fails the unsupported one. Its aggregate pass rate is $0.9$ , but its unsupported-case pass rate is $0$ . The contract-aware replay passes all ten. These are deterministic candidate response fixtures, not outputs from a live LLM or a model-quality benchmark. They let us test whether the evaluator and release policy catch a known defect before spending money on generation.

Trace enough to diagnose, without retaining everything

A trace connects one request’s spans: retrieval, model invocation, tool calls, validation and response. Keep request IDs, version IDs, timing, error category and useful aggregate diagnostics. Raw prompts, retrieved documents and tool outputs can contain private data; define redaction, access and retention before collecting them. This lab records fixture IDs and summary scores in two actual MLflow traces. It does not log private messages. Tracing observes behavior; it does not enforce tool permissions or protect against prompt injection.

Add the production cases deliberately

Extend the suite with missing/contradictory context, multilingual inputs, long contexts, prompt injection in retrieved text, malformed tool arguments, unsupported tools, timeouts and rate limits. Treat retrieved text as data, not trusted instructions. Check tool allowlists and permissions outside the model. Keep a held-out test set and human-reviewed examples separate from cases used to tune prompts. If using an LLM judge, record the judge version and rubric and compare with human labels; judge disagreement is evidence to inspect, not an infallible score.

Measure serving work in the right units

Measure prefill and decode separately, along with time to first token, output tokens per second, completion rate, tail latency and request cost under declared context lengths and concurrency. Provider token accounting must use the provider’s tokenizer and pricing; whitespace counts are not interchangeable tokens. Set retry and timeout budgets and test the fallback path. The text harness supplies actual tiny-model cache/export measurements; the replay lesson supplies evaluation/tracing contracts, not production throughput.

Run the complete local prompt/trace demonstration

bash

# Run run the complete local prompt/trace demonstration using the course Python environment
python projects/engineering-release/mlflow_lab.py --output ./engineering-run-llm

Expected:

Registers a real prompt version and records a nested retrieval/replay trace; no paid model API is used.

A missing response is an evaluation failure

Freeze the case inventory before collecting responses. Pair responses by stable case ID, not by arrival order: concurrent requests may finish out of order, and a timeout may omit a reply entirely. A loop over paired lists can silently stop at the shorter list and make coverage disappear from the denominator. The new experiment accepts reordered complete replies but rejects missing, duplicate and unknown IDs. In a live application, retain a timeout or tool-error outcome under the original case ID and score it under a declared policy; do not delete difficult cases after seeing results.

The unsupported case changes the release decision

Predict before looking: How can a candidate pass 90% overall and still fail every unsupported case?

Always-answer replay passes nine ordinary cases but fails the unsupported case; the contract-aware replay passes all ten. — Plot from a recorded CPU computation · JAX 0.9.2 · 1 CPU device

Read the figure

The horizontal axis names the aggregate, answerable and unsupported groups. The vertical axis is the deterministic fixture pass fraction, between $0$ and $1$ . Always-answer scores $0.9,1,0$ ; contract-aware scores $1,1,1$ . The aggregate has ten cases, answerable has nine, and unsupported has one. The two answerable bars have equal height because both replays contain the same correct answers there.

Connect it to the computation

The plot reports evaluator outputs from controlled answer objects recorded in real traces. It does not report accuracy of a live language model. In a real suite, expand critical cases and preserve sample counts and model-version evidence before making a release claim.

Open full-size figure Download data and provenance

The notebook contains the data experiment, plotting code, and recorded figure output. Rerun it to test your changes.

Where to put & run this code: Work in phases/16-operations/07-llmops-prompts-evaluation-and-traces/code/main.py (or downloaded exercises/operations-07.py / exercises/operations-07.ipynb) and execute from the repository root with python3 phases/16-operations/07-llmops-prompts-evaluation-and-traces/code/main.py.

1 Prepare the explicit contract

Create main.py and add this setup block. Continue with the next block in the same file.

main.py · append build step 1

# Step 1 — Prepare the explicit contract: For a text generator, record the model revision, tokenizer, prompt...
import os
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_DISABLE_AGENT_HINT"]="1"
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_ENABLE_ASYNC_TRACE_LOGGING"]="false"
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_ENABLE_TELEMETRY"]="false"
# Import required JAX, NumPy, and standard-library modules.
import tempfile
from pathlib import Path
import mlflow
from mlflow import MlflowClient

2 Run and inspect the controlled experiment

Append this block, run main.py in the course environment, and retain the actual output.

main.py · append build step 2

# Step 2 — Run and inspect the controlled experiment: A trace connects one request’s spans: retrieval, model invocation,...
def llm_case(case, answer, allowed_citations):
    """Deterministic replay evaluator, not an LLM judge or a general safety classifier."""
    if set(answer) != {'answer', 'citations', 'abstain'} or not isinstance(answer['answer'], str):
        return False
    if not isinstance(answer['citations'], list) or any(not isinstance(c, str) for c in answer['citations']):
        return False
    if type(answer['abstain']) is not bool or not set(answer['citations']) <= set(allowed_citations):
        return False
    if case['unanswerable']:
        return answer['abstain'] and not answer['citations'] and answer['answer'] == ''
    return not answer['abstain'] and answer['answer'] == case['expected'] and case['document'] in answer['citations']

cases = [dict(id=f'answerable-{i}', unanswerable=False, expected='CPU', document='course-v1') for i in range(9)]
# Accumulate the next contribution into `cases`.
cases += [dict(id='unsupported', unanswerable=True)]
answer = dict(answer='CPU', citations=['course-v1'], abstain=False)
unsafe = [dict(answer) for _ in cases]
guarded = [dict(answer) for _ in cases[:-1]] + [dict(answer='', citations=[], abstain=True)]
results = []
# Create an isolated temporary directory to run and inspect artifacts safely:
with tempfile.TemporaryDirectory(prefix='llmops-') as folder:
    # Execute `mlflow.set_tracking_uri('sqlite:///' + str(Path(folder) / 'traces.db'))`.
    mlflow.set_tracking_uri('sqlite:///' + str(Path(folder) / 'traces.db'))
    client = MlflowClient()
    experiment = client.create_experiment('replay-evaluation', artifact_location=(Path(folder)/'artifacts').as_uri())
    mlflow.set_experiment(experiment_id=experiment)
    for name,replies in [('always-answer',unsafe),('contract-aware',guarded)]:
        with mlflow.start_span(name='evaluate-replay', span_type='CHAIN') as span:
            span.set_inputs({'candidate': name, 'dataset_version': 'public-fixture-v1', 'case_count': len(cases)})
            passed = [llm_case(case, reply, ['course-v1']) for case,reply in zip(cases,replies)]
            summary = {'overall': sum(passed)/len(passed), 'answerable': sum(passed[:-1])/9, 'unanswerable': float(passed[-1])}
            span.set_outputs(summary)
            results.append(summary)
    traces = mlflow.search_traces(experiment_ids=[experiment], return_type='list')
    assert len(traces) == 2
assert results[0] == {'overall': .9, 'answerable': 1., 'unanswerable': 0.}
assert results[1] == {'overall': 1., 'answerable': 1., 'unanswerable': 1.}
assert not llm_case(cases[0],dict(answer='CPU',citations=['invented'],abstain=False),['course-v1'])
print('Observed replay scores:', results)
print('Two real MLflow traces; no LLM API was called and no general model quality was evaluated.')

Code building blocks in this step

llm_case(...) — Call llm_case with your updated parameters or inputs from this lesson's workspace.
isinstance(...) — Call isinstance with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

3 Step 3: Verify invariants on the completed state

Run the final shape and numerical assertions to confirm the state built in Steps 1 and 2.

main.py · append build step 3

assert results[1] == {'overall': 1., 'answerable': 1., 'unanswerable': 1.}
assert not llm_case(cases[0],dict(answer='CPU',citations=['invented'],abstain=False),['course-v1'])

Code building blocks in this step

llm_case(...) — Call llm_case with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

Checking these invariants confirms the computation is ready for the full worked experiment.

View the cumulative build in main.py

These blocks append in order to the same file.

main.py · cumulative Python build

# Step 1 — Prepare the explicit contract: For a text generator, record the model revision, tokenizer, prompt...
import os
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_DISABLE_AGENT_HINT"]="1"
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_ENABLE_ASYNC_TRACE_LOGGING"]="false"
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_ENABLE_TELEMETRY"]="false"
# Import required JAX, NumPy, and standard-library modules.
import tempfile
from pathlib import Path
import mlflow
from mlflow import MlflowClient

# Step 2 — Run and inspect the controlled experiment: A trace connects one request’s spans: retrieval, model invocation,...
def llm_case(case, answer, allowed_citations):
    """Deterministic replay evaluator, not an LLM judge or a general safety classifier."""
    if set(answer) != {'answer', 'citations', 'abstain'} or not isinstance(answer['answer'], str):
        return False
    if not isinstance(answer['citations'], list) or any(not isinstance(c, str) for c in answer['citations']):
        return False
    if type(answer['abstain']) is not bool or not set(answer['citations']) <= set(allowed_citations):
        return False
    if case['unanswerable']:
        return answer['abstain'] and not answer['citations'] and answer['answer'] == ''
    return not answer['abstain'] and answer['answer'] == case['expected'] and case['document'] in answer['citations']

cases = [dict(id=f'answerable-{i}', unanswerable=False, expected='CPU', document='course-v1') for i in range(9)]
# Accumulate the next contribution into `cases`.
cases += [dict(id='unsupported', unanswerable=True)]
answer = dict(answer='CPU', citations=['course-v1'], abstain=False)
unsafe = [dict(answer) for _ in cases]
guarded = [dict(answer) for _ in cases[:-1]] + [dict(answer='', citations=[], abstain=True)]
results = []
# Create an isolated temporary directory to run and inspect artifacts safely:
with tempfile.TemporaryDirectory(prefix='llmops-') as folder:
    # Execute `mlflow.set_tracking_uri('sqlite:///' + str(Path(folder) / 'traces.db'))`.
    mlflow.set_tracking_uri('sqlite:///' + str(Path(folder) / 'traces.db'))
    client = MlflowClient()
    experiment = client.create_experiment('replay-evaluation', artifact_location=(Path(folder)/'artifacts').as_uri())
    mlflow.set_experiment(experiment_id=experiment)
    for name,replies in [('always-answer',unsafe),('contract-aware',guarded)]:
        with mlflow.start_span(name='evaluate-replay', span_type='CHAIN') as span:
            span.set_inputs({'candidate': name, 'dataset_version': 'public-fixture-v1', 'case_count': len(cases)})
            passed = [llm_case(case, reply, ['course-v1']) for case,reply in zip(cases,replies)]
            summary = {'overall': sum(passed)/len(passed), 'answerable': sum(passed[:-1])/9, 'unanswerable': float(passed[-1])}
            span.set_outputs(summary)
            results.append(summary)
    traces = mlflow.search_traces(experiment_ids=[experiment], return_type='list')
    assert len(traces) == 2
assert results[0] == {'overall': .9, 'answerable': 1., 'unanswerable': 0.}
assert results[1] == {'overall': 1., 'answerable': 1., 'unanswerable': 1.}
assert not llm_case(cases[0],dict(answer='CPU',citations=['invented'],abstain=False),['course-v1'])
print('Observed replay scores:', results)
print('Two real MLflow traces; no LLM API was called and no general model quality was evaluated.')

assert results[1] == {'overall': 1., 'answerable': 1., 'unanswerable': 1.}
assert not llm_case(cases[0],dict(answer='CPU',citations=['invented'],abstain=False),['course-v1'])

Run the example

python3 phases/16-operations/07-llmops-prompts-evaluation-and-traces/code/main.py

About this command

This shortcut uses an activated macOS/Linux course environment in the repository folder. For Windows, use the explicit interpreter command in “Setup help” above. The script includes worked examples and reference solutions.

Inspect the complete assembled example

main.py · complete Python example

# Step 1 — Prepare the explicit contract: For a text generator, record the model revision, tokenizer, prompt...
import os
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_DISABLE_AGENT_HINT"]="1"
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_ENABLE_ASYNC_TRACE_LOGGING"]="false"
# Configure environment variable before initializing the runtime.
os.environ["MLFLOW_ENABLE_TELEMETRY"]="false"
# Import required JAX, NumPy, and standard-library modules.
import tempfile
from pathlib import Path
import mlflow
from mlflow import MlflowClient

# Step 2 — Run and inspect the controlled experiment: A trace connects one request’s spans: retrieval, model invocation,...
def llm_case(case, answer, allowed_citations):
    """Deterministic replay evaluator, not an LLM judge or a general safety classifier."""
    if set(answer) != {'answer', 'citations', 'abstain'} or not isinstance(answer['answer'], str):
        return False
    if not isinstance(answer['citations'], list) or any(not isinstance(c, str) for c in answer['citations']):
        return False
    if type(answer['abstain']) is not bool or not set(answer['citations']) <= set(allowed_citations):
        return False
    if case['unanswerable']:
        return answer['abstain'] and not answer['citations'] and answer['answer'] == ''
    return not answer['abstain'] and answer['answer'] == case['expected'] and case['document'] in answer['citations']

cases = [dict(id=f'answerable-{i}', unanswerable=False, expected='CPU', document='course-v1') for i in range(9)]
# Accumulate the next contribution into `cases`.
cases += [dict(id='unsupported', unanswerable=True)]
answer = dict(answer='CPU', citations=['course-v1'], abstain=False)
unsafe = [dict(answer) for _ in cases]
guarded = [dict(answer) for _ in cases[:-1]] + [dict(answer='', citations=[], abstain=True)]
results = []
# Create an isolated temporary directory to run and inspect artifacts safely:
with tempfile.TemporaryDirectory(prefix='llmops-') as folder:
    # Execute `mlflow.set_tracking_uri('sqlite:///' + str(Path(folder) / 'traces.db'))`.
    mlflow.set_tracking_uri('sqlite:///' + str(Path(folder) / 'traces.db'))
    client = MlflowClient()
    experiment = client.create_experiment('replay-evaluation', artifact_location=(Path(folder)/'artifacts').as_uri())
    mlflow.set_experiment(experiment_id=experiment)
    for name,replies in [('always-answer',unsafe),('contract-aware',guarded)]:
        with mlflow.start_span(name='evaluate-replay', span_type='CHAIN') as span:
            span.set_inputs({'candidate': name, 'dataset_version': 'public-fixture-v1', 'case_count': len(cases)})
            passed = [llm_case(case, reply, ['course-v1']) for case,reply in zip(cases,replies)]
            summary = {'overall': sum(passed)/len(passed), 'answerable': sum(passed[:-1])/9, 'unanswerable': float(passed[-1])}
            span.set_outputs(summary)
            results.append(summary)
    traces = mlflow.search_traces(experiment_ids=[experiment], return_type='list')
    assert len(traces) == 2
assert results[0] == {'overall': .9, 'answerable': 1., 'unanswerable': 0.}
assert results[1] == {'overall': 1., 'answerable': 1., 'unanswerable': 1.}
assert not llm_case(cases[0],dict(answer='CPU',citations=['invented'],abstain=False),['course-v1'])
print('Observed replay scores:', results)
print('Two real MLflow traces; no LLM API was called and no general model quality was evaluated.')

Expected result: Observed replay scores: [{'overall': 0.9, 'answerable': 1.0, 'unanswerable': 0.0}, {'overall': 1.0, 'answerable': 1.0, 'unanswerable': 1.0}] Two real MLflow traces; no LLM API was called and no general model quality was evaluated.

Recorded reference execution

Reference output captured from executing the complete lesson script (code/main.py) and its assertions in the course CPU environment.

2026-10-20T10:51:53.818799+00:00 · Python 3.14.3 · JAX 0.9.2 · 1 CPU device

Observed replay scores: [{'overall': 0.9, 'answerable': 1.0, 'unanswerable': 0.0}, {'overall': 1.0, 'answerable': 1.0, 'unanswerable': 1.0}]
Two real MLflow traces; no LLM API was called and no general model quality was evaluated.
Observed replay scores: [{'overall': 0.9, 'answerable': 1.0, 'unanswerable': 0.0}, {'overall': 1.0, 'answerable': 1.0, 'unanswerable': 1.0}]
Two real MLflow traces; no LLM API was called and no general model quality was evaluated.
Release decisions: [False, True]
Complete inventory: {'count': 10, 'passed': 9, 'fraction': 0.9} ; missing, duplicate and unknown responses rejected.
Unrecognized citations rejected.
An abstention flag alone does not satisfy the output contract.
Failed answerable work remains in denominator: {'count': 10, 'passed': 9, 'fraction': 0.9}
PASS: operations-07

Download execution record · The notebook includes executed cells and figure outputs. Compare your run with these values; timings depend on the machine.

Separate aggregate and critical-case behavior

Separate aggregate and critical-case behavior · Python experiment

# Experiment — Separate aggregate and critical-case behavior: The extra gate encodes a required behavior that the overall...
release=[r['overall']>=.9 and r['unanswerable']==1. for r in results]
assert release==[False,True]
print('Release decisions:',release)

Expected: Release decisions are False and True.

The extra gate encodes a required behavior that the overall average hides. One unsupported fixture is a regression test, not a population estimate.

Reject an incomplete replay before reporting a score

Reject an incomplete replay before reporting a score · Python experiment

# Experiment — Reject an incomplete replay before reporting a score: The case inventory defines the denominator.
def score_replay(inventory, replies, allowed_citations):
    ids = [case['id'] for case in inventory]
    reply_ids = [reply['id'] for reply in replies]
    # Guard input contract (`not ids or len(set(ids)) != len(ids)`) and fail fast if violated.
    if not ids or len(set(ids)) != len(ids):
        raise ValueError('empty or duplicate case inventory')
    # Guard input contract (`len(set(reply_ids)) != len(reply_ids) or set(reply_ids) != set(ids)`) and fail fast if violated.
    if len(set(reply_ids)) != len(reply_ids) or set(reply_ids) != set(ids):
        raise ValueError('missing, duplicate or unknown response IDs')
    by_id = {reply['id']: reply['response'] for reply in replies}
    passed = [llm_case(case, by_id[case['id']], allowed_citations) for case in inventory]
    return {'count': len(ids), 'passed': sum(passed), 'fraction': sum(passed) / len(ids)}

complete_replies = [{'id': case['id'], 'response': reply} for case, reply in zip(cases, unsafe)]
replay_summary = score_replay(cases, list(reversed(complete_replies)), ['course-v1'])
assert replay_summary == {'count': 10, 'passed': 9, 'fraction': .9}
for broken in (complete_replies[:-1], complete_replies + [complete_replies[0]],
               complete_replies[:-1] + [dict(id='unknown', response=answer)]):
    try:
        score_replay(cases, broken, ['course-v1'])
    except ValueError:
        pass
    else:
        raise AssertionError('incomplete or ambiguous evaluation accepted')
print('Complete inventory:', replay_summary, '; missing, duplicate and unknown responses rejected.')

Expected: Ten cases, nine passes, fraction 0.9; reordering preserves the score. Missing, duplicate and unknown response identities are rejected.

The case inventory defines the denominator. In particular, losing the unsupported case must not convert a critical failure into a perfect answerable-only score. This checks evaluation bookkeeping on replayed data, not live generation quality.

Hands-On Exercise · Modify & Verify Run in the declared lesson environment

Make it yours

Where to put your code: Append your solution at the bottom of phases/16-operations/07-llmops-prompts-evaluation-and-traces/code/main.py (or inside the Make it yours cell of exercises/operations-07.ipynb) and run python3 phases/16-operations/07-llmops-prompts-evaluation-and-traces/code/main.py to verify your prediction before opening the reference solution.

Add a response with an invented citation and require the evaluator to fail it.

Show a writing guide and starter scaffold

How to write this exercise code: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

llm_case(...) — Call llm_case with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

2. Step-by-step implementation plan:

Set up the changed input or configuration parameters requested in the prompt.
Run the target function or transformation and assert that the output matches your prediction.

Starter code scaffold · copy into your file & fill in the TODOs

# Exercise solution: Add a response with an invented citation and require the evaluator to...
for citation in ['invented','old-index-v0']:
    assert not llm_case(cases[0],dict(answer='CPU',citations=[citation],abstain=False),['course-v1'])  # TODO: complete assertion check
print('Unrecognized citations rejected.')

Reveal the reference solution

Reference solution

# Exercise solution: Add a response with an invented citation and require the evaluator to...
for citation in ['invented','old-index-v0']:
    assert not llm_case(cases[0],dict(answer='CPU',citations=[citation],abstain=False),['course-v1'])
print('Unrecognized citations rejected.')

Check abstention structure

transfer

Try a response that claims to abstain but still returns an unsupported answer.

Show a writing guide and starter scaffold

How to write this practice solution: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

structure(...) — Call structure with your updated parameters or inputs from this lesson's workspace.
llm_case(...) — Call llm_case with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

2. Step-by-step implementation plan:

Check abstention structure (transfer): Validate the full response object; a boolean alone cannot...

Starter code scaffold · copy into your file & fill in the TODOs

# Check abstention structure (transfer): Validate the full response object; a boolean alone cannot...
bad = dict(...)  # TODO: compute bad
assert not llm_case(cases[-1],bad,['course-v1'])  # TODO: complete assertion check
print('An abstention flag alone does not satisfy the output contract.')

Show a hint

The fixture contract requires both empty answer text and no citations.

Reveal reference solution and reasoning

Check abstention structure · Python solution

# Check abstention structure (transfer): Validate the full response object; a boolean alone cannot...
bad=dict(answer='An unsupported claim',citations=[],abstain=True)
assert not llm_case(cases[-1],bad,['course-v1'])
print('An abstention flag alone does not satisfy the output contract.')

Validate the full response object; a boolean alone is distinct from that the application withheld an answer.

Keep failed work in the evaluation inventory

Transfer

Replace the answer for the first answerable case with an empty abstention. Keep the case ID and show how the complete-suite score changes.

Show a writing guide and starter scaffold

How to write this practice solution: step-by-step recipe & starter scaffold

1. Key functions & syntax to use:

inventory(...) — Call inventory with your updated parameters or inputs from this lesson's workspace.
zip(...) — Call zip with your updated parameters or inputs from this lesson's workspace.
assert condition — Verify that the observed output shape, status, or numerical value satisfies the contract.

2. Step-by-step implementation plan:

Set up the changed input or configuration parameters requested in the prompt.
Run the target function or transformation and assert that the output matches your prediction.

Starter code scaffold · copy into your file & fill in the TODOs

# Keep failed work in the evaluation inventory (Transfer): The fixture represents an unsuccessful answer as an empty...
timeout_replies = ...  # TODO: compute timeout_replies
timeout_replies[0]['response'] = dict(...)  # TODO: compute timeout_replies[0]['response']
timeout_summary = score_replay(...)  # TODO: compute timeout_summary
assert timeout_summary  # TODO: complete assertion check
print('Failed answerable work remains in denominator:', timeout_summary)

Show a hint

Use the contract-aware replies as the starting point. An empty abstention is correct for the unsupported case but wrong for the selected answerable case.

Reveal reference solution and reasoning

Keep failed work in the evaluation inventory · Python solution

# Keep failed work in the evaluation inventory (Transfer): The fixture represents an unsuccessful answer as an empty...
timeout_replies = [{'id': case['id'], 'response': dict(reply)} for case, reply in zip(cases, guarded)]
timeout_replies[0]['response'] = dict(answer='', citations=[], abstain=True)
timeout_summary = score_replay(cases, timeout_replies, ['course-v1'])
assert timeout_summary == {'count': 10, 'passed': 9, 'fraction': .9}
print('Failed answerable work remains in denominator:', timeout_summary)

The fixture represents an unsuccessful answer as an empty abstention. A real trace should additionally distinguish timeout, retrieval failure and deliberate abstention as causes, while retaining all cases in the predeclared metric.

Diagnose the result

When quality changes, compare prompt/model/retrieval/tool versions and case-level traces before tuning weights. A high average can hide an unsupported-case regression. Do not call a synthetic replay a measured LLM response.

Carry forward

Version the application boundary
Trace enough to diagnose, without retaining everything
Measure serving work in the right units

Keep your evidence

Download evidence template

Keep all case IDs and per-slice replay scores, missing/duplicate-response rejections, the retained failed case, trace identities and application-version fields. Explain what requires separate live-model evaluation.

Save your modified code, its output, and an explanation of your prediction. Checkpoint answers and exercise evidence are tracked separately.

Saved in this browser. Export or back up your portfolio anytime in My learning.

Save work for this lesson

Verified on local CPU. Stage to a Cloud TPU VM via Cloud TPU setup to record accelerator execution receipts (TODO: add automated Cloud TPU CI execution receipts).

LLMOps: version prompts, evaluate behavior and trace requests

What you will be able to do

The problem

The idea

Trace a response through the system that produced it

Version the application boundary

Define cases before changing the prompt

Read the rare-case failure

Trace enough to diagnose, without retaining everything

Add the production cases deliberately

Measure serving work in the right units

Run the complete local prompt/trace demonstration

A missing response is an evaluation failure

The unsupported case changes the release decision

Read the figure

Connect it to the computation

1 Prepare the explicit contract

2 Run and inspect the controlled experiment

3 Step 3: Verify invariants on the completed state

Run the example

Recorded reference execution

Separate aggregate and critical-case behavior

Reject an incomplete replay before reporting a score

Make it yours

Check abstention structure

Keep failed work in the evaluation inventory

Check your understanding

Diagnose the result

Carry forward

Keep your evidence

Primary references