{ "version": "https://jsonfeed.org/version/1.1", "title": "ReactorJet", "home_page_url": "https://reactorjet.com/", "feed_url": "https://reactorjet.com/rss/feed.json", "description": "Notes from the lab. Two series: control loop (how a system converges) and teardown (how a system fails the closure test).", "language": "en", "items": [ { "id": "https://reactorjet.com/notes/control-loop/agent-activity-is-not-progress", "url": "https://reactorjet.com/notes/control-loop/agent-activity-is-not-progress", "title": "Agent activity is not progress", "summary": "Tool calls and busy run logs measure agent activity; progress begins when the workflow can prove that the target state moved closer and stayed there.", "content_text": "Agent dashboards are very good at showing motion. Runs started. Tools called.\nTokens spent. Files changed. Tests executed. A stream of events makes the system\nlook productive even when the original problem is still open.\n\nActivity is an operating cost. It is not evidence of progress.\n\nThis distinction sounds obvious until a run gets stuck. An agent edits a test,\nruns it, reads the failure, edits another file, and repeats. The event count\nrises. The transcript becomes impressive. If the user-visible defect remains,\nthe workflow has produced activity and no progress.\n\n## Progress needs a target state\n\nYou cannot measure progress from the agent's behavior alone. You need a target\nstate and evidence that compares it with what existed before.\n\nFor a coding task, the target may include a specific behavior in the deployed\napplication, not merely a patch and a green local suite. For an incident agent,\nprogress might mean the error rate has returned below a threshold and remained\nthere for an observation window. For a support agent, drafting twelve replies is\nactivity; resolving the right cases under the applicable policy is progress.\n\nThe unit should be a verified state transition. \"Opened pull request\" can be a\nuseful workflow milestone, but it does not mean \"fixed the problem.\" \"Sent\nresponse\" does not mean \"resolved the request.\" The names on the dashboard\nshould preserve those differences.\n\nThis is where many agent metrics go wrong. They measure what is easy to count\ninside the run and ignore what happened to the system outside it. A model can\noptimize tool use without improving the outcome. It can also reduce tool calls\nby skipping checks, which looks efficient until a bad change escapes.\n\n## Busy loops need an exit condition\n\nAn agent should not be allowed to turn uncertainty into indefinite work. Every\nloop needs a condition for success, a reason to continue, and a point where it\nstops consuming resources and asks for help.\n\nConsider an agent investigating a slow service. Querying logs is reasonable at\nthe start. After six similar queries produce no new distinction between likely\ncauses, another query is not progress. The next useful action may be to request\nmissing telemetry, run a bounded experiment, or hand the case to someone with\ndifferent access.\n\nA retry budget helps, but a raw attempt count is crude. Track whether each pass\nproduced new evidence, eliminated a plausible cause, or changed the candidate\nsolution. Repetition without information gain should make the workflow stop\nsooner than a difficult investigation that is still narrowing the problem.\n\n## Exploration can still count\n\nNot all valuable work closes a task immediately. Research, diagnosis, and design\noften begin without a known route to the answer. Forcing them into a simplistic\ncompletion metric encourages premature certainty.\n\nExploration can count as progress when it leaves a durable change in what the\nteam knows. A failed experiment may rule out an approach. A trace may reveal\nwhich boundary lacks telemetry. A prototype may show that the requested latency\nis incompatible with a dependency.\n\nThe evidence still matters. \"Investigated several options\" is activity. \"Option\nB failed the saved workload because it exceeded the agreed limit\" is a result\nsomeone else can use. The second statement reduces uncertainty even though no\ncode shipped.\n\nActivity metrics remain useful for cost and diagnosis. Tool-call counts can\nexpose a loop. Runtime can show a slow dependency. Token use can help compare\nworkflow designs. Just do not promote those numbers into outcome metrics. A\ncheaper run that fails more often is not automatically better, and a longer run\nthat closes a hard case is not automatically worse.\n\n## Change the scoreboard\n\nPick one agent workflow and write its target state in observable terms. Add a\ncompletion check that runs outside the agent's narrative, then label intermediate\nevents for what they are: attempted, drafted, tested, reviewed, or applied. Do\nnot call the work complete until the target-state evidence exists.\n\nFor runs that do not close, record the useful delta: evidence gathered, a cause\nexcluded, or a precise blocker with an owner. Put activity metrics on a separate\ncost view. If the same run looks productive only when the event stream is on\nscreen, the scoreboard is measuring theatre.", "date_published": "2026-08-13T00:00:00.000Z", "date_modified": "2026-08-13T00:00:00.000Z", "authors": [ { "name": "tim" } ], "tags": [ "agent-systems", "evals", "autonomic-swe" ] }, { "id": "https://reactorjet.com/notes/control-loop/approval-fatigue-is-a-workflow-design-problem", "url": "https://reactorjet.com/notes/control-loop/approval-fatigue-is-a-workflow-design-problem", "title": "Approval fatigue is a workflow design problem", "summary": "When an agent asks for approval too often and without decision-ready context, the workflow trains people to click through instead of protecting the actions that deserve judgment.", "content_text": "Approval fatigue is usually blamed on the reviewer. They were distracted. They\nstopped reading. They approved something they should have challenged.\n\nThat diagnosis is convenient and mostly wrong. If a workflow asks a person to\napprove routine, low-consequence steps all day, it is teaching them that the\napproval button is part of operating the agent. The same button later appears\nfor a production migration or a public message, but the reviewer has already\nlearned the gesture: click so the work can continue.\n\nThe problem is not that people need more discipline. The workflow has failed to\nseparate ordinary execution from consequential judgment.\n\n## An approval is not a remote-control button\n\nImagine a coding agent that asks before reading a log, installing an existing\ndependency, running a test, formatting a file, and opening a pull request. Each\nrequest is technically honest. Together they make the person act as a slow\nsubprocess scheduler.\n\nThen the agent asks to change an authorization rule. That request deserves a\ndifferent kind of attention, but it arrives in the same shape as the formatting\nrequest. The interface has flattened unlike risks into identical prompts.\n\nA useful approval should mark a boundary. It should tell the reviewer why this\nstep needs authority the agent does not have. It should also contain enough\ncontext to make a decision without reconstructing the whole run.\n\n\"Allow command?\" is not enough. A decision-ready request explains:\n\n- what will change and what will remain untouched;\n- which evidence supports the change;\n- who or what could be affected;\n- whether the action can be reversed, and how;\n- what the agent will do if approval is denied or expires.\n\nIf the workflow cannot provide that information, it is asking for permission,\nnot informed approval.\n\n## Reduce approvals without hiding risk\n\nThe answer is not to make the agent autonomous everywhere. It is to spend human\nattention where judgment changes the outcome.\n\nStart by classifying actions according to consequence and reversibility. Reading\na repository and running a protected test suite may be allowed within a bounded\nworkspace. Editing files can remain private until the proposed patch is ready\nfor review. Deploying, spending money, changing identity or access, and sending\nexternal communication can retain stronger gates.\n\nThe exact boundary depends on the system. A healthcare workflow and an internal\nprototype should not share an approval policy. What matters is that the policy\nis explicit and tied to risk, rather than inherited from whichever tool happens\nto be called.\n\nBatching can help, but only when the grouped actions belong to one coherent\ndecision. Approving \"apply this patch, run these checks, and open a draft pull\nrequest\" may be reasonable because the work stays reviewable. Bundling a data\nmigration with an unrelated public announcement saves clicks by concealing two\ndifferent consequences. That is worse than fatigue.\n\nGood workflows also remember bounded decisions. If a reviewer has approved a\nspecific test command for a named repository and branch, the agent should not\nask again five minutes later unless the scope changed. Approval should not\nsilently become permanent access, but it should survive long enough to complete\nthe decision that was actually made.\n\n## Some friction is doing real work\n\nThere are cases where repeated approval is appropriate. A regulated process may\nrequire a fresh signer at each controlled transition. An incident responder may\nwant every destructive command held while the system is unstable. A novel task\nmay be too poorly understood for a broad permission grant.\n\nKeep that friction. Make its purpose visible. A reviewer is more likely to pay\nattention when the request says which policy requires the approval and what\nevidence will be recorded.\n\nThe test is not whether the workflow has fewer prompts. The test is whether a\nperson can still recognize the prompt that needs thought.\n\n## Fix one approval queue\n\nReview the last twenty approval requests from one recurring agent workflow. For\neach request, write down the consequence, reversibility, and information the\nreviewer received. Remove repeated prompts for bounded low-risk actions. Split\nrequests that hide separate decisions. Rewrite the remaining prompts as small\ndecision packets with evidence and an undo path.\n\nThen watch the next run. If the reviewer still has to reconstruct context or\napprove the same boundary twice, the workflow is not finished.", "date_published": "2026-08-13T00:00:00.000Z", "date_modified": "2026-08-13T00:00:00.000Z", "authors": [ { "name": "tim" } ], "tags": [ "agent-systems", "trust-boundary", "runtime-infra" ] }, { "id": "https://reactorjet.com/notes/control-loop/rollback-belongs-in-the-plan", "url": "https://reactorjet.com/notes/control-loop/rollback-belongs-in-the-plan", "title": "Rollback belongs in the plan", "summary": "An agent should know how a consequential action will be contained or reversed before it acts, because a rollback invented after failure is only another untested production change.", "content_text": "Teams often ask for a rollback plan immediately after something goes wrong.\nThat is several hours too late.\n\nOnce an agent has deployed a broken change, deleted data, or sent a message, the\nsystem is already in a state the original plan did not cover. Any rollback\ninvented at that point is a fresh production action, designed under pressure and\naimed at a system that may still be moving.\n\nRollback belongs beside the proposed action, before approval and before apply.\n\n## \"We can revert the commit\" is not a plan\n\nA commit can be reverted while its effects remain. A deployment may have run a\ndatabase migration. A worker may have emitted jobs that are still queued. A\nconfiguration change may have altered traffic before the old version returns.\nCustomers may already have received an email that cannot be unsent.\n\nThe plan has to describe the state transition as well as the file change.\n\nSuppose an agent proposes a new billing rule. The forward plan says which code\nwill deploy and which tests pass. A real rollback plan also asks whether invoices\nwill be written under the new rule, how those records will be identified, and\nwhether restoring the old code can interpret them. If the answer is \"we will\nwork that out if needed,\" the system is not ready for unattended promotion.\n\nFor each consequential action, the agent should be able to state:\n\n- the signal that will trigger a stop or reversal;\n- the authority allowed to make that decision;\n- the mechanism that restores a known state;\n- the data or side effects that the mechanism cannot restore;\n- the check that proves recovery actually happened.\n\nThis information changes the forward plan. A dangerous migration may become a\nstaged dual-write. A broad release may become a small traffic slice. A public\nmessage may remain a draft until a person approves it. Rollback design is not\nadministrative paperwork after engineering. It shapes safer engineering.\n\n## Some actions need compensation, not rollback\n\nNot every effect is reversible. Sent messages, disclosed information, completed\npayments, and many physical actions cannot be returned to their previous state.\nCalling the response a rollback creates false confidence.\n\nThese workflows need a compensation plan. That could mean issuing a correcting\nmessage, creating an offsetting transaction, revoking access, or escalating to\nan incident owner. Compensation does not erase the original action. It limits\nfurther harm and establishes what happens next.\n\nThe distinction matters when assigning authority. An agent may be allowed to\nroll back a canary deployment automatically after a protected health check\nfails. It should not invent and send an apology to customers merely because a\nmetric crossed a threshold. The second action carries new reputational and legal\nconsequences.\n\n## Rollback itself can be unsafe\n\nReturning to an older version is not always the lowest-risk move. The old binary\nmay be incompatible with newly written data. A security fix may not be safe to\nremove. During an incident, repeated automatic reversals can make diagnosis\nharder and turn a small fault into oscillation.\n\nA good plan can therefore choose to halt, isolate, or roll forward instead. The\nrequirement is not \"always revert.\" The requirement is to decide the containment\nstrategy while there is still time to test it.\n\nThat strategy should have limits. Set a retry budget. Name the state that counts\nas recovered. Record when automatic recovery must stop and hand control to a\nperson. An agent that keeps applying and reverting because each local check\nlooks plausible is not resilient. It is thrashing.\n\n## Plan the way back before the next change\n\nTake the next agent action that can affect shared state. Before it runs, write\nthe failure signal, containment mechanism, decision authority, and recovery\ncheck in the same proposal as the forward change. Test the mechanism against a\nstaging state that resembles the one it would have to recover.\n\nIf an effect cannot be reversed, say so plainly and require a compensation path\nor a smaller initial action. Do not approve the forward plan until the way back,\nor the reason there is no way back, is visible.", "date_published": "2026-08-13T00:00:00.000Z", "date_modified": "2026-08-13T00:00:00.000Z", "authors": [ { "name": "tim" } ], "tags": [ "autonomic-swe", "agent-systems", "runtime-infra", "trust-boundary" ] }, { "id": "https://reactorjet.com/notes/control-loop/your-agent-is-creating-evidence-debt", "url": "https://reactorjet.com/notes/control-loop/your-agent-is-creating-evidence-debt", "title": "Your agent is creating evidence debt", "summary": "When an agent changes a system faster than you preserve why its work was accepted, each successful run leaves future operators with uncertainty they will have to repay.", "content_text": "An agent completes a task, reports that the checks passed, and moves on. The\nchange works. Nobody asks for more detail.\n\nThree weeks later, another person needs to understand why the change was safe.\nThe run log is gone, the screenshot has no build identifier, and the summary\nsays only that all tests passed. The patch is still there, but the reason it was\naccepted has disappeared.\n\nThat is evidence debt. The workflow took the benefit of a fast decision without\npaying the cost of making that decision inspectable later.\n\n## Logs are not automatically evidence\n\nAgent systems produce plenty of material: tool calls, token traces, terminal\noutput, screenshots, model explanations, and chat transcripts. Volume can hide\nthe problem. A pile of run data is not the same as evidence for a decision.\n\nEvidence has to answer a question. Which version was tested? What behavior was\nobserved? Which policy applied? Who or what accepted the result? Can the result\nbe tied to the exact change that reached production?\n\nConsider a deployment where the agent records a green test run and a screenshot\nof the homepage. If the tests ran against one commit while the screenshot came\nfrom an older deployment, both artifacts are real and the conclusion is false.\nWithout identity and provenance, evidence can decorate a decision without\nsupporting it.\n\nAgent-written summaries need the same skepticism. A summary is useful for\nnavigation, but it is still a claim made by the system that performed the work.\n\"No regressions found\" should point to the checks, comparison, and scope that\njustify it. Otherwise the next operator has to repeat the investigation.\n\n## Debt compounds when the system changes\n\nWeak evidence may seem harmless while the original operator remembers the run.\nThat memory acts as an undocumented index. It disappears when the person changes\nteams, the model changes, or several similar runs blur together.\n\nThe cost surfaces during a rollback, an audit, or a regression investigation.\nPeople rerun old tests against new dependencies. They infer intent from code.\nThey compare screenshots by eye and hope the environments match. Work that felt\nfast at acceptance becomes expensive when the decision has to be reconstructed\nunder pressure.\n\nThe debt also limits learning. If a team cannot connect accepted changes to\ntheir evidence, it cannot tell which checks predict good outcomes and which\nones merely produce green marks. Every evaluation starts from anecdotes.\n\n## Preserve a receipt, not the whole run\n\nThe answer is not indefinite storage of every token and shell line. Most raw\ntraces are noisy, expensive, and full of context that should not be retained.\nSome contain secrets or personal data. Evidence collection needs limits too.\n\nA durable receipt should be small enough to inspect and specific enough to\nreplay. For a software change, it might identify the input task, proposed diff,\ntested revision, check results, decision, and rollback reference. A customer\nsupport workflow might retain the policy version and approval record without\nkeeping unrelated customer data in the agent's reasoning trace.\n\nMatch the receipt to the lifespan and blast radius of the action. A private\ndraft that will be discarded tomorrow does not need the same record as a schema\nmigration. A reversible internal change may need less than a payment or a public\nclaim. The point is not universal retention. It is deliberate retention.\n\nThe receipt also needs an owner. If nobody is responsible for whether it remains\nreadable and linked to the change, the storage bucket becomes another form of\ndebt.\n\n## Pay at the point of acceptance\n\nChoose one recurring agent action that can affect production or another person.\nDefine the minimum receipt required before that action is accepted. Make the\nworkflow capture identifiers and evidence automatically where it can, then ask\nthe approving authority to record the decision rather than relying on the\nagent's recap.\n\nTest the receipt by giving it to someone who did not watch the run. They should\nbe able to identify what changed, why it passed, and where to start if it must be\nreversed. Any question that sends them back to a vanished chat is unpaid debt.", "date_published": "2026-08-13T00:00:00.000Z", "date_modified": "2026-08-13T00:00:00.000Z", "authors": [ { "name": "tim" } ], "tags": [ "agent-systems", "evals", "trust-boundary" ] }, { "id": "https://reactorjet.com/notes/control-loop/a-passing-test-is-not-a-good-result", "url": "https://reactorjet.com/notes/control-loop/a-passing-test-is-not-a-good-result", "title": "A passing test is not always a good result", "summary": "Tests tell an agent whether known checks passed; they do not automatically prove that the change helped users, respected every boundary, or is safe to keep.", "content_text": "An agent edits a file, runs the test suite, and gets a green result. Is the work\ndone?\n\nSometimes. But a passing test only answers the questions encoded in that test.\nIt does not tell you whether the agent solved the right problem, introduced a\nnew risk outside the test suite, or changed the checks to make its own work look\nbetter.\n\nThis distinction matters anywhere an agent can both change a system and read\nthe result. The test output is useful feedback. It becomes a trustworthy\ndecision only when the rules for passing are protected from the thing being\njudged.\n\n## Feedback and decisions are different\n\nAn agent needs feedback while it works. Compiler errors, unit tests, screenshots,\nand logs all help it improve a proposal. Let the agent see them and respond.\n\nThe final decision is different. A decision answers a more consequential\nquestion: should this change be kept, promoted, deployed, or allowed to affect\nsomeone else?\n\nThat decision should use evidence the agent cannot quietly weaken. Depending on\nthe risk, that might mean:\n\n- held-out examples the agent cannot edit;\n- a policy check running with separate credentials;\n- a comparison with the current baseline, not just an absolute score;\n- a required human review for security, money, identity, or irreversible work;\n- an automatic stop when previously passing behavior regresses.\n\n## Four questions to ask\n\nYou do not need a new platform to improve an agent workflow. Start with four\nquestions:\n\n1. **What exactly can the agent change?** A patch, a database record, a message,\n or a deployment should have a visible boundary.\n2. **What evidence says the result is better?** Name the test, comparison,\n policy, or human judgment rather than saying the result “looks good.”\n3. **Can the agent change that evidence?** If it can edit the tests, examples,\n or scoring rules that judge it, separate those permissions.\n4. **What happens when the evidence is missing or unclear?** A safe workflow\n waits or stops. It does not silently treat uncertainty as success.\n\nThese questions apply whether you are building an agent, buying one, or simply\ndeciding whether to trust what it produced.\n\n## A useful first improvement\n\nChoose one agent task you already run. Save a small set of real examples before\nchanging the prompt or model. Run the current and proposed versions against the\nsame examples. Record regressions as well as improvements, and do not let the\nagent remove a difficult example during the run.\n\nThat is not a complete evaluation system. It is enough to establish the most\nimportant rule: the agent can propose a change, but it does not get to rewrite\nthe meaning of success while being judged.\n\n## Continue\n\n- [Use the four-part closure test](/reference/closure-test)\n- [See the two-part architecture](/reference/two-plane-loop)\n- [Inspect the agent-eval toolkit](/packs/autonomic-swe/agent-eval)", "date_published": "2026-08-12T00:00:00.000Z", "date_modified": "2026-08-12T00:00:00.000Z", "authors": [ { "name": "tim" } ], "tags": [ "agent-systems", "evals", "trust-boundary" ] }, { "id": "https://reactorjet.com/notes/control-loop/your-agent-should-not-be-its-own-reviewer", "url": "https://reactorjet.com/notes/control-loop/your-agent-should-not-be-its-own-reviewer", "title": "Your agent should not be its own reviewer", "summary": "Let an agent explain and improve its work, but use separate evidence and authority for the decision to accept, publish, merge, or deploy it.", "content_text": "Agents are often asked to review their own work. The pattern feels efficient:\ngenerate an answer, critique it, revise it, and stop when the same model says it\nis good enough.\n\nSelf-review is useful for iteration. It is weak as the final authority.\n\nThe agent shares the assumptions, blind spots, context, and incentives that\nproduced the first answer. A second pass can catch obvious mistakes, but it\ndoes not create an independent reason to trust the result.\n\n## Keep the useful part of self-review\n\nLet the agent inspect its draft. Ask it to find missing evidence, test edge\ncases, and explain uncertainty. This can improve quality before a final check\nand reduce avoidable review work.\n\nThen separate the decision that has consequences:\n\n- the coding agent may run tests, but a protected job decides whether the pull\n request can merge;\n- the support agent may draft a refund, but a policy service decides whether\n the amount is permitted;\n- the research agent may summarize sources, but citation checks use the source\n documents rather than the agent's confidence;\n- the publishing agent may prepare a post, but a person approves claims that\n affect reputation or legal exposure.\n\nThe goal is not to add a human to every step. It is to give the final decision\nan authority that is not controlled by the proposal being judged.\n\n## Match independence to the risk\n\nNot every agent action needs the same level of separation. A private draft can\nuse lightweight checks. A production deployment, payment, account change, or\npublic claim needs stronger evidence and a clear handoff.\n\nA practical scale is:\n\n1. **Draft:** the agent acts in a private workspace and explains its work.\n2. **Checked:** automated tests or policies inspect the result.\n3. **Compared:** the result is measured against a saved baseline or held-out\n examples.\n4. **Approved:** a separate service or person authorizes the consequential\n action.\n\nMove up the scale as the blast radius grows. Make the level visible so users\nknow whether they are looking at a suggestion, a checked result, or an approved\naction.\n\n## The design rule\n\nThe agent may help produce evidence. It may not be the only authority deciding\nwhat that evidence means.\n\nThat one rule makes agent workflows easier to debug, easier to explain, and\nsafer to expand over time. It also lets teams change models without rebuilding\nthe meaning of success around every new model.\n\n## Continue\n\n- [Read the starting guide](/notes/control-loop/a-passing-test-is-not-a-good-result)\n- [Learn the two-part architecture](/reference/two-plane-loop)\n- [See how Cursor performs against the same questions](/notes/teardown/scoring-cursor)", "date_published": "2026-08-11T00:00:00.000Z", "date_modified": "2026-08-11T00:00:00.000Z", "authors": [ { "name": "tim" } ], "tags": [ "agent-systems", "evals", "trust-boundary" ] }, { "id": "https://reactorjet.com/notes/control-loop/closing-the-loop", "url": "https://reactorjet.com/notes/control-loop/closing-the-loop", "title": "Build the checks before you build the agent", "summary": "Decide how you will judge the result before building the agent, or your examples and tests will slowly start to favor whatever the agent already does well.", "content_text": "The order is not negotiable, and it is the opposite of what feels productive.\n\nWhen you are starting a new autonomic system, the natural instinct is to build\nthe Proposer first. The Proposer is the most interesting component. It is the\nthing that takes a task and emits a change. It is where the magic is. Of\ncourse you start there.\n\nYou will regret it. Here is what happens:\n\n1. You build a Proposer. It is good. It emits patches that are usually right.\n2. You build a Scorer to evaluate the Proposer's output.\n3. The Scorer uses a corpus. You assemble the corpus from cases that the\n Proposer tends to do well on, because those are the cases you noticed.\n4. The Scorer reports a high number. The Proposer is good. You ship it.\n5. Six weeks later, the Proposer regresses on a case that wasn't in the\n corpus. The number is still high. You don't notice.\n6. The number is high because the corpus was shaped to flatter the Proposer,\n not because the Proposer is good.\n\nThis is the recurring finding of the teardown series. The system that fails\ncondition 4 fails it in exactly this way: the judge is on the same team as\nthe proposer.\n\n## A safer build order\n\nBuild, in this order, with no skipping:\n\n1. **Verdict plane.** The corpus, the rubric, the scorer, the plane layout,\n the separation guard. Nothing else. At this stage there is no loop; there\n is a way to score a change a human made. If you cannot tell whether a\n change helped, that is the thing to build, and it is also the thing that\n will still be true in a year when the Proposer has been replaced twice.\n\n2. **Ledger.** Append-only, before there is anything to record. Retrofitting\n an audit trail onto a running loop produces an audit trail with a gap\n exactly where it matters.\n\n3. **Sensors and the quoting function.** The trust boundary goes in before\n the first untrusted byte, not after the first incident.\n\n4. **Applier.** Idempotent apply, discard, promote-with-token. Test discard\n harder than apply.\n\n5. **Proposer.** Last. It is the most interesting component and the most\n replaceable, and building it first is how the corpus ends up shaped to\n flatter it.\n\n## What \"build the checks first\" actually means\n\nIt means: before you write a line of agent code, you write the test that\ndecides whether a change helped. The test is held by a separate authority\nfrom the agent. The agent submits to it; it does not call it.\n\nA practical starting point: take a small corpus of cases that you can score\nby hand. Run your future Proposer against the corpus. Score the output\nyourself. Notice which cases the Proposer gets wrong that you would have\ngotten right. Add those cases to the corpus. Repeat until the Proposer\neither improves or the corpus reveals that the Proposer is the wrong shape\nfor the task.\n\nThis is slow. It is also the only way to build a corpus that discriminates\nrather than flatters.\n\n## See also\n\n- [two-plane-loop](/reference/two-plane-loop): the architecture\n- [closure-test](/reference/closure-test): the rubric, especially condition 4", "date_published": "2026-08-06T00:00:00.000Z", "date_modified": "2026-08-06T00:00:00.000Z", "authors": [ { "name": "tim" } ], "tags": [ "autonomic-swe", "agent-systems" ] }, { "id": "https://reactorjet.com/notes/teardown/scoring-cursor", "url": "https://reactorjet.com/notes/teardown/scoring-cursor", "title": "Can Cursor prove its own work?", "summary": "Cursor can write code, run tests, and iterate, but a person still approves the work and the same agent that makes the change also interprets whether it worked.", "content_text": "Cursor is the most-deployed agent IDE in current use. It writes code, runs\ntests, observes failures, and iterates. The surface looks like MAPE-K. The\nquestion is whether it actually closes the loop, or whether the loop is open\nin a way that matters.\n\n## What Cursor does\n\n- A user prompt becomes a multi-step agent run.\n- The agent edits files, runs the project's test suite, reads the output, and\n iterates.\n- The agent can use tools: `read_file`, `edit_file`, `terminal`, `web_search`.\n- The user sees a diff and either accepts or rejects.\n\n## Four practical questions\n\n| Condition | Pass? | Notes |\n|---|---|---|\n| 1. Closed loop | partial | Cursor observes the test result, but the observation is in-process and not recorded anywhere durable. The \"loop\" is the agent's scratch state, not a Ledger. |\n| 2. Declarative target | yes | The target is the diff. `git diff` after the agent run shows the full intent. The transcript is also available, but the diff is the source of truth. |\n| 3. Unattended in the ordinary case | no | Cursor requires a human at every turn. The \"yolo mode\" (auto-accept) shifts the human to the end, not out of the loop. |\n| 4. Ground truth | partial | The agent runs the user's tests, but the user wrote the tests. The eval is whatever the user said it was. There is no separate Verdict plane; the Proposer (the agent) calls the tests and reads the result in the same process. |\n\n## Where the workflow still needs help\n\n**Condition 3 is the loudest failure.** The whole point of an agent IDE is\nthat the human types a prompt and the IDE does the work. But the human is still\nin the loop, approving each change. The system is not unattended in the\nordinary case; it is attended in *every* case.\n\n**Condition 4 is the more interesting failure.** The \"ground truth\" is the\ntest suite, but the test suite is not a separate authority. The agent that\nproposes the change also runs the eval and reads the result. There is no plane\nseparation. A change that \"passes\" the tests passed because the agent's\nproposal and the test runner are in the same process. The trust boundary\nlives in the user's trust, not in the architecture.\n\nA subtler version of the same problem: Cursor's iteration is bounded by what\nfits in the context window and by the user's patience. The \"verdict\" is the\nagent's decision to stop iterating, which is the same agent that decided to\nkeep iterating. KILL — the case that was passing and now fails — is not a\nverdict the agent emits. It is a fact the user discovers when they look at\nthe diff.\n\n## What would make the workflow stronger\n\nTo satisfy condition 3, Cursor would need an unattended mode with a named\nexception class. The exception would be \"tests do not exist\" or \"the user is\nin the file\" or something specific. The current \"yolo mode\" does not have a\nnamed exception class; it has a duration limit.\n\nTo satisfy condition 4, Cursor would need to separate the Proposer from the\nVerifier. A change would land in a scratch workspace, an independent Scorer\nwould run a held-out corpus against the change, and a PromotionToken would\ngate the merge. The current architecture has no such plane; the Proposer is\nalso the Verifier.\n\n## See also\n\n- [closure-test](/reference/closure-test) — the rubric\n- [autonomic-software-engineering](/reference/autonomic-software-engineering) — the practice", "date_published": "2026-08-06T00:00:00.000Z", "date_modified": "2026-08-06T00:00:00.000Z", "authors": [ { "name": "tim" } ], "tags": [ "autonomic-swe", "agent-systems", "evals" ] } ] }