An agent edits a file, runs the test suite, and gets a green result. Is the work
done?

Sometimes. But a passing test only answers the questions encoded in that test.
It does not tell you whether the agent solved the right problem, introduced a
new risk outside the test suite, or changed the checks to make its own work look
better.

This distinction matters anywhere an agent can both change a system and read
the result. The test output is useful feedback. It becomes a trustworthy
decision only when the rules for passing are protected from the thing being
judged.

## Feedback and decisions are different

An agent needs feedback while it works. Compiler errors, unit tests, screenshots,
and logs all help it improve a proposal. Let the agent see them and respond.

The final decision is different. A decision answers a more consequential
question: should this change be kept, promoted, deployed, or allowed to affect
someone else?

That decision should use evidence the agent cannot quietly weaken. Depending on
the risk, that might mean:

- held-out examples the agent cannot edit;
- a policy check running with separate credentials;
- a comparison with the current baseline, not just an absolute score;
- a required human review for security, money, identity, or irreversible work;
- an automatic stop when previously passing behavior regresses.

## Four questions to ask

You do not need a new platform to improve an agent workflow. Start with four
questions:

1. **What exactly can the agent change?** A patch, a database record, a message,
   or a deployment should have a visible boundary.
2. **What evidence says the result is better?** Name the test, comparison,
   policy, or human judgment rather than saying the result “looks good.”
3. **Can the agent change that evidence?** If it can edit the tests, examples,
   or scoring rules that judge it, separate those permissions.
4. **What happens when the evidence is missing or unclear?** A safe workflow
   waits or stops. It does not silently treat uncertainty as success.

These questions apply whether you are building an agent, buying one, or simply
deciding whether to trust what it produced.

## A useful first improvement

Choose one agent task you already run. Save a small set of real examples before
changing the prompt or model. Run the current and proposed versions against the
same examples. Record regressions as well as improvements, and do not let the
agent remove a difficult example during the run.

That is not a complete evaluation system. It is enough to establish the most
important rule: the agent can propose a change, but it does not get to rewrite
the meaning of success while being judged.

## Continue

- [Use the four-part closure test](/reference/closure-test)
- [See the two-part architecture](/reference/two-plane-loop)
- [Inspect the agent-eval toolkit](/packs/autonomic-swe/agent-eval)