When an agent run fails, the last error is often only where the failure became
visible. A malformed tool response may have entered the run several steps
earlier. A check may have used the wrong baseline. An action may have succeeded
even though the runner timed out.

Trace the saved run before rerunning it. A rerun can be useful later, but it
creates a new sequence with different timing and possibly different external
state.

## 1. Freeze the failed run

Start with the run identifier and prevent further automatic retries while you
inspect it. Preserve the workspace, proposal, logs, check output, and current
target state according to your retention rules.

Record the debugging question in one sentence:

```text
Why did RUN-8894 end with STOP after the agent produced a valid configuration?
```

Then capture the run's final known state:

```text
run_id: RUN-8894
task_id: TASK-1061
runner_state: stopped
proposal_ref: sha256:...
verdict: STOP
live_action: not attempted
workspace: retained
```

Do not edit the original records while investigating. Add annotations in a
separate trace or incident record.

Expected outcome: everyone investigating refers to the same failed execution,
not a later retry or a reconstructed transcript.

## 2. Collect records by identifier

Gather records that share `run_id`, `task_id`, `proposal_ref`, approval ID, or
action attempt ID. At minimum, look for:

- the task contract and its version;
- the baseline and checker versions;
- observations supplied to the agent;
- tool requests and responses;
- agent attempts and budget events;
- the final proposal or partial workspace;
- check results and raw evidence;
- approval requests and decisions;
- live action attempts, if any;
- cleanup, rollback, or reconciliation results.

Secrets and raw personal data should remain redacted according to policy. Keep
stable references to protected records so an authorized investigator can
inspect them when necessary.

Missing records are findings. Write down what is absent instead of filling the
gap with the agent's summary.

## 3. Build a factual timeline

Order events using sequence numbers when available. Clocks across services can
drift, so timestamps alone may put cause after effect.

| Sequence | Component | Event | Evidence |
|---|---|---|---|
| 41 | runner | accepted task version 8 | `task/TASK-1061/8` |
| 42 | sensor | read current retry policy | `observation/991` |
| 43 | agent | produced proposal | `sha256:...` |
| 44 | checker | loaded baseline version 11 | `baseline/11` |
| 45 | checker | protected retry case failed | `check/774/raw` |
| 46 | verdict | emitted `STOP` | `verdict/310` |
| 47 | applier | discarded workspace changes | `cleanup/558` |

Use the timeline to separate facts from interpretations. "Protected retry case
failed" is a fact when the raw check result supports it. "The agent ignored the
policy" is an interpretation until you identify which input, proposal field,
or tool result demonstrates that.

Expected outcome: the trace shows the first observed divergence rather than
only the final error message.

## 4. Check record integrity and linkage

Before diagnosing behavior, confirm that the records belong together:

1. Does the proposal hash match the content the checker read?
2. Did the checker use the baseline named in the run contract?
3. Do tool responses link to the request and attempt that produced them?
4. Was the verdict computed from the recorded check results?
5. If an approval existed, did it name the same proposal and target?
6. Does cleanup refer to the workspace created for this run?

If a link is wrong, stop interpreting the agent's reasoning. You have a trace
or orchestration defect first. Repairing the prompt will not fix a checker that
read the wrong proposal.

## 5. Find the first broken assumption

Walk forward from the task contract and ask what each component assumed about
its input. Compare that assumption with the saved record.

Common categories are:

| Category | Evidence to look for | Typical recovery |
|---|---|---|
| Task defect | Ambiguous objective, missing constraint, stale input version | Correct and version the task, then start a new run |
| Agent defect | Proposal contradicts a supplied constraint or omits required output | Revise instructions, tools, or examples and rerun the eval set |
| Tool defect | Timeout, truncated response, malformed payload, wrong target | Fix or isolate the tool, then retry from a known state |
| Check defect | Wrong baseline, unstable case, unreadable evidence | Repair the checker and recompute the verdict without changing the proposal |
| Boundary defect | Missing, overbroad, expired, or mismatched approval | Refuse execution and issue a new scoped request |
| Target defect | External system rejected or partly accepted the action | Reconcile target state before deciding whether to retry |
| Runtime defect | Budget, storage, credential, or worker failure | Restore the runtime and resume only from a documented safe point |

Choose the earliest category supported by evidence. Later errors may be real,
but fixing only the final symptom can leave the original defect in place.

In the example timeline, inspect why the protected retry case failed. If the
proposal violates a policy present in the task input, the agent produced a bad
proposal and `STOP` is correct. If the checker loaded baseline 11 while the run
contract names baseline 12, the verdict is not trustworthy even though the
case did fail.

## 6. Verify the safety state

Before planning a retry, answer these questions:

- Did any proposed change reach the live target?
- Is the scratch workspace isolated or fully discarded?
- Was an approval token issued, consumed, or left valid?
- Can an automatic retry still start?
- Does the target need reconciliation because the action result is uncertain?

Do not infer live state from the runner's status. A runner can report a timeout
after the target accepted an action. Query the target through an independent
read path and save the result.

Expected outcome: the trace states whether the failure is contained, rolled
back, partially applied, or unresolved.

## 7. Write the failure report

Keep the report short enough to use during repair, but link every conclusion to
evidence.

```text
run_id: RUN-8894
classification: check configuration defect
first_divergence: checker loaded baseline 11 instead of required baseline 12
evidence:
  - task/TASK-1061/8
  - check/774/config
impact: no live action attempted; proposal discarded
recovery: fix baseline resolution and recompute checks for the unchanged proposal
prevention: reject checker startup when the requested baseline cannot be loaded
owner: verdict-plane operator
status: contained
```

Avoid conclusions such as "the model was confused" unless you can tie them to
a specific decision and input. Even then, describe the observable mismatch.
That gives the next builder something they can test.

## Check the trace

Before closing the investigation, verify both the explanation and the safety
claim:

- A second person can follow the evidence links and reach the same first
  divergence.
- The report distinguishes missing evidence from evidence of absence.
- The proposed recovery addresses the first broken assumption.
- A negative test reproduces the failure or proves the new guard rejects it.
- The live target and any approval token have a recorded final state.
- The failed run remains available after the repair and is linked to the new
  test case.

## Handle incomplete traces

If a tool response is missing, classify the run as unresolved rather than
inventing its contents. Improve the record path before increasing the agent's
authority.

If sequence order is ambiguous, state the competing explanations and collect
the evidence that would distinguish them. Do not choose the most convenient
story because it fits the final error.

If the target state cannot be read, block retries that could duplicate the
action. Escalate to the owner of that system with the action attempt ID and the
last confirmed state.

If the failure cannot be reproduced, keep the original evidence and add
instrumentation around the suspected boundary. A clean rerun does not erase a
failed run.

## Next steps

- [Run an agent task with an independent check](/guides/tutorial/run-an-agent-task-with-an-independent-check)
- [Read the two-plane loop reference](/reference/two-plane-loop)
- [Use the closure test to assess the repaired workflow](/reference/closure-test)