Skip to content
Hardenedpack0.2.0

agent-eval

Closes the loop on agent evaluation by separating the Verdict plane from the Proposal plane, so a change to the corpus can never silently tune the judge that decides whether the change helped.

Maintained by Tim
  • autonomic-swe
  • evals
  • agent-systems

Install

$ npx @reactorjet/pack install autonomic-swe/agent-eval

Download

Tarball
0.2.0.tar.gz
SHA-256
c492fce8c9f2202d5b44b431c0ae1e5bd00b0043588fb65082330e62d9781368
Size
235.1 KB

The checksum detects corruption, not tampering — artifacts are not signed yet.

Evals

path
tests/
count
504
threshold
100%

Failure modes

  • 1a case that was passing in the baseline is now failing on a new model version

    recoveryVerdictComputer emits KILL (not PASS) for any per-case regression; the regressed case is recorded in `regressions` and the loop halts promotion

  • 2LLM judge non-determinism masquerading as improvement (same input, different scores run-to-run)

    recoveryn-run comparison with effect-size threshold; below the declared minimum the verdict is STALL, not PASS, with the iteration count recorded

  • 3Verdict plane unreachable (scorer, corpus, or ledger failure)

    recoveryloop halts, no promotion, outage recorded; absence of a verdict is never treated as a pass (blueprint invariant 8: fail closed)

  • 4Proposer stuck in a doom loop (iteration budget exhausted without movement)

    recoverySTALL verdict is emitted with the iteration count and reason; no silent stop, no default to pass

  • 5Sensor returns untrusted content that looks like instructions (prompt-injection-shaped)

    recovery`quote()` wraps the content in delimiters and redacts instruction-shaped lines without echoing the original bytes; the QuotedContext records the quoting method for the Ledger

Trust boundary

Untrusted input enters via Sensor.observe() (issue text, test output, trace data, file contents). All such data is typed as an Observation. The Observation constructor is private to the Sensor interface — consumers cannot fabricate trusted-looking Observations. Observations reach prompt context only through `quote()`, which wraps the content in delimiters, redacts instruction-shaped lines (without echoing the original bytes), and records the quoting method in the QuotedContext. The Verdict plane (Scorer, Corpus, Ledger, VerdictComputer, assert_plane_separation) is mechanically separated from the Proposal plane; the Proposal plane cannot write any path the Verdict plane reads, enforced at the commit hook and the credential.

Version

License
MIT
Verified on claude-sonnet 4.6 on 2026-08-01
claude-sonnet 4.6

agent-eval is the reference pack for the practice the lab is named after. It is the Verdict plane: the part of the two-plane loop that decides whether a change helped, without being the part that proposed the change. The two-plane-loop blueprint’s §9 build order is Verdict plane first, Proposer last, and the Verdict plane is what this pack ships.

What it satisfies, in the closure test

The closure test (brief section 1a) is four conditions; no maturity levels. This pack satisfies three of the four directly, and ships the machinery for the fourth to be satisfied by deployment choice.

  • Condition 1 — Closed loop. The Applier writes to a workspace; the Sensors observe the result; the Ledger records both. A campaign is n runs of the same case set against baseline and new revisions, with the Ledger appending a snapshot per run per config.
  • Condition 2 — Declarative target. A Proposal is a content-addressed patch; intent is the diff, not a transcript. git diff (or the equivalent on the managed subject) is the full intent.
  • Condition 3 — Unattended in the ordinary case. The blast-radius machinery is here (L0 Shadow through L3 Self-modifying), and a deployment that runs at L2+ is unattended in the ordinary case with named exception classes. Whether this deployment is unattended is a choice — the default is L0.
  • Condition 4 — Ground truth. The Verdict plane has plane separation (mechanical, not procedural), a monotone corpus (35+ tasks, cases may be quarantined but not removed), and n-run comparison with effect-size threshold. A float does not decide; a categorical verdict does.

The single architectural rule that holds the rest up: the thing that proposes the change does not decide whether the change worked. The Verdict plane decides. The Proposer submits to it.

What it does not ship

  • A real Proposer. Per the blueprint’s §9 build order, the Proposer is built last because building it first is how the corpus ends up shaped to flatter it. The NoOpProposer is honest about being a stub.
  • An L2+ deployment. The blast-radius profiles are machinery; the deployment choice is the operator’s. L0 (Shadow) is the safe default until the corpus has been observed to discriminate.
  • The L3 frozen-verdict-plane discipline. That requirement exists (L3_SELF_MODIFYING requires the verdict plane to be version-frozen for the duration of a campaign) but no implementation enforces it. A loop that can change both the proposer and the scorer within one campaign has no fixed point and its history is uninterpretable; do not run L3 yet.

Install

uv sync
uv run agent-eval --help
uv run pytest                  # 504 tests; must all pass

The pack is consumable as a Python library (from agent_eval.verdict import VerdictComputer) and as a CLI (agent-eval tasks, agent-eval coverage, agent-eval history). MCP transport is a target but not yet implemented.

Keep exploring

Use the toolkit with the bigger picture in view

Starting guide

A passing test is not always a good result

See why an agent needs more than a green check before its work is accepted.

Architecture

Keep the agent separate from the judge

Understand the simple design rule behind the pack.

Source code

Inspect the repository

Review the implementation and its test suite at the pinned source.