Independent evaluator

Purpose

An independent evaluator judges a proposed result without relying on the producer’s confidence. It compares the intent brief, the subject artifact, and the evidence packet with a versioned rubric. It returns a verdict for each criterion and for the proposal as a whole.

Independence is a control relationship, not a job title. The producer should not be able to change the rubric, select only favorable evidence, overwrite the verdict, or grant the capability that follows a passing result.

When to use

Use an independent evaluator when a workflow needs more than self-review before it can merge, publish, deploy, or hand work to another system. It is especially useful when the worker is probabilistic, the acceptance criteria combine several checks, or a passing test can still miss the requested outcome.

The evaluator can be deterministic code, a separate model invocation, a test harness, a human review role, or a combination. Choose the simplest form that can judge the stated criteria. Use deterministic checks for facts that can be computed directly. Reserve model or human judgment for criteria that require interpretation.

Inputs and outputs

Inputs:

• the exact intent brief version;
• the proposed artifact or change digest;
• a sealed evidence packet;
• a versioned rubric with required and advisory criteria;
• evaluator identity and execution environment;
• saved examples or baselines when the rubric requires comparison.

Outputs:

• a verdict of pass, fail, insufficient-evidence, or error;
• one result per rubric criterion;
• evidence references and reasons for every result;
• the evaluator and rubric versions;
• recommended next action, limited to retry, revise, review, or stop.

Smallest useful interface

type CriterionResult = {
  criterionId: string;
  result: "pass" | "fail" | "insufficient-evidence" | "error";
  evidenceRefs: string[];
  reason: string;
};

type Evaluation = {
  subjectDigest: string;
  evaluatorVersion: string;
  rubricVersion: string;
  verdict: "pass" | "fail" | "insufficient-evidence" | "error";
  criteria: CriterionResult[];
};

function evaluate(input: {
  intentRef: string;
  evidencePacketRef: string;
  rubricRef: string;
}): Evaluation;

The evaluator should reject a packet whose subject digest does not match the proposal. A required criterion without evidence should not pass by default.

Example

Rubric checkout-timeout@2 contains two required criteria:

1. The timeout path returns the documented retry response.
2. Existing successful checkout behavior has regression evidence.

The evidence packet contains a passing focused timeout test but says the full integration suite could not run. The evaluator returns:

{
  "subjectDigest": "sha256:41de...",
  "evaluatorVersion": "checkout-evaluator@4",
  "rubricVersion": "checkout-timeout@2",
  "verdict": "insufficient-evidence",
  "criteria": [
    {
      "criterionId": "timeout-response",
      "result": "pass",
      "evidenceRefs": ["packet:run-1842#timeout-test"],
      "reason": "The focused test exercises the required timeout response and exits successfully."
    },
    {
      "criterionId": "successful-checkout-regression",
      "result": "insufficient-evidence",
      "evidenceRefs": [],
      "reason": "The required integration check did not run."
    }
  ]
}

The worker may repair the test environment or add valid regression evidence. It may not convert the second criterion to advisory during the run.

Failure modes

• The producer also owns the rubric. Criteria can drift toward the current output. Put rubric changes behind separate review and version them.
• The evaluator reads the worker’s summary instead of source evidence. A persuasive explanation can replace observation. Resolve evidence references and judge the artifacts directly.
• Missing evidence becomes a pass. Use a distinct insufficient-evidence result so absence is not confused with success.
• The evaluator changes between retries without a record. Results cannot be compared. Store the evaluator version, rubric version, and subject digest.
• A single score hides a required failure. Report criterion-level results and define which criteria are mandatory.
• The worker retries until a variable judge passes. Set retry limits, retain every verdict, and route repeated disagreement to review.
• The evaluator can execute the proposed change. Keep judgment separate from the capability to merge, deploy, publish, or spend.
• Saved examples only cover familiar successes. Add boundary cases and known failures when maintaining the rubric. Do not let the producer choose the evaluation set for each run.

What it does not solve

An independent evaluator does not make a weak rubric meaningful. It cannot detect conditions that the evidence and criteria never expose. Separation also does not guarantee different blind spots if the producer and evaluator use the same inputs and assumptions. The component creates a reviewable authority boundary; teams still need to test the evaluator and update its rubric through a controlled process.

Related reading

• Evidence packet
• Human approval checkpoint
• Policy gate
• Your agent should not be its own reviewer
• Closure test