Trace a failed agent run

When an agent run fails, the last error is often only where the failure became visible. A malformed tool response may have entered the run several steps earlier. A check may have used the wrong baseline. An action may have succeeded even though the runner timed out.

Trace the saved run before rerunning it. A rerun can be useful later, but it creates a new sequence with different timing and possibly different external state.

1. Freeze the failed run

Start with the run identifier and prevent further automatic retries while you inspect it. Preserve the workspace, proposal, logs, check output, and current target state according to your retention rules.

Record the debugging question in one sentence:

Why did RUN-8894 end with STOP after the agent produced a valid configuration?

Then capture the run’s final known state:

run_id: RUN-8894
task_id: TASK-1061
runner_state: stopped
proposal_ref: sha256:...
verdict: STOP
live_action: not attempted
workspace: retained

Do not edit the original records while investigating. Add annotations in a separate trace or incident record.

Expected outcome: everyone investigating refers to the same failed execution, not a later retry or a reconstructed transcript.

2. Collect records by identifier

Gather records that share run_id, task_id, proposal_ref, approval ID, or action attempt ID. At minimum, look for:

• the task contract and its version;
• the baseline and checker versions;
• observations supplied to the agent;
• tool requests and responses;
• agent attempts and budget events;
• the final proposal or partial workspace;
• check results and raw evidence;
• approval requests and decisions;
• live action attempts, if any;
• cleanup, rollback, or reconciliation results.

Secrets and raw personal data should remain redacted according to policy. Keep stable references to protected records so an authorized investigator can inspect them when necessary.

Missing records are findings. Write down what is absent instead of filling the gap with the agent’s summary.

3. Build a factual timeline

Order events using sequence numbers when available. Clocks across services can drift, so timestamps alone may put cause after effect.

Sequence	Component	Event	Evidence
41	runner	accepted task version 8	task/TASK-1061/8
42	sensor	read current retry policy	observation/991
43	agent	produced proposal	sha256:...
44	checker	loaded baseline version 11	baseline/11
45	checker	protected retry case failed	check/774/raw
46	verdict	emitted STOP	verdict/310
47	applier	discarded workspace changes	cleanup/558

Use the timeline to separate facts from interpretations. “Protected retry case failed” is a fact when the raw check result supports it. “The agent ignored the policy” is an interpretation until you identify which input, proposal field, or tool result demonstrates that.

Expected outcome: the trace shows the first observed divergence rather than only the final error message.

4. Check record integrity and linkage

Before diagnosing behavior, confirm that the records belong together:

1. Does the proposal hash match the content the checker read?
2. Did the checker use the baseline named in the run contract?
3. Do tool responses link to the request and attempt that produced them?
4. Was the verdict computed from the recorded check results?
5. If an approval existed, did it name the same proposal and target?
6. Does cleanup refer to the workspace created for this run?

If a link is wrong, stop interpreting the agent’s reasoning. You have a trace or orchestration defect first. Repairing the prompt will not fix a checker that read the wrong proposal.

5. Find the first broken assumption

Walk forward from the task contract and ask what each component assumed about its input. Compare that assumption with the saved record.

Common categories are:

Category	Evidence to look for	Typical recovery
Task defect	Ambiguous objective, missing constraint, stale input version	Correct and version the task, then start a new run
Agent defect	Proposal contradicts a supplied constraint or omits required output	Revise instructions, tools, or examples and rerun the eval set
Tool defect	Timeout, truncated response, malformed payload, wrong target	Fix or isolate the tool, then retry from a known state
Check defect	Wrong baseline, unstable case, unreadable evidence	Repair the checker and recompute the verdict without changing the proposal
Boundary defect	Missing, overbroad, expired, or mismatched approval	Refuse execution and issue a new scoped request
Target defect	External system rejected or partly accepted the action	Reconcile target state before deciding whether to retry
Runtime defect	Budget, storage, credential, or worker failure	Restore the runtime and resume only from a documented safe point

Choose the earliest category supported by evidence. Later errors may be real, but fixing only the final symptom can leave the original defect in place.

In the example timeline, inspect why the protected retry case failed. If the proposal violates a policy present in the task input, the agent produced a bad proposal and STOP is correct. If the checker loaded baseline 11 while the run contract names baseline 12, the verdict is not trustworthy even though the case did fail.

6. Verify the safety state

Before planning a retry, answer these questions:

• Did any proposed change reach the live target?
• Is the scratch workspace isolated or fully discarded?
• Was an approval token issued, consumed, or left valid?
• Can an automatic retry still start?
• Does the target need reconciliation because the action result is uncertain?

Do not infer live state from the runner’s status. A runner can report a timeout after the target accepted an action. Query the target through an independent read path and save the result.

Expected outcome: the trace states whether the failure is contained, rolled back, partially applied, or unresolved.

7. Write the failure report

Keep the report short enough to use during repair, but link every conclusion to evidence.

run_id: RUN-8894
classification: check configuration defect
first_divergence: checker loaded baseline 11 instead of required baseline 12
evidence:
  - task/TASK-1061/8
  - check/774/config
impact: no live action attempted; proposal discarded
recovery: fix baseline resolution and recompute checks for the unchanged proposal
prevention: reject checker startup when the requested baseline cannot be loaded
owner: verdict-plane operator
status: contained

Avoid conclusions such as “the model was confused” unless you can tie them to a specific decision and input. Even then, describe the observable mismatch. That gives the next builder something they can test.

Check the trace

Before closing the investigation, verify both the explanation and the safety claim:

• A second person can follow the evidence links and reach the same first divergence.
• The report distinguishes missing evidence from evidence of absence.
• The proposed recovery addresses the first broken assumption.
• A negative test reproduces the failure or proves the new guard rejects it.
• The live target and any approval token have a recorded final state.
• The failed run remains available after the repair and is linked to the new test case.

Handle incomplete traces

If a tool response is missing, classify the run as unresolved rather than inventing its contents. Improve the record path before increasing the agent’s authority.

If sequence order is ambiguous, state the competing explanations and collect the evidence that would distinguish them. Do not choose the most convenient story because it fits the final error.

If the target state cannot be read, block retries that could duplicate the action. Escalate to the owner of that system with the action attempt ID and the last confirmed state.

If the failure cannot be reproduced, keep the original evidence and add instrumentation around the suspected boundary. A clean rerun does not erase a failed run.

Next steps

• Run an agent task with an independent check
• Read the two-plane loop reference
• Use the closure test to assess the repaired workflow