Build an immutable coding-agent evaluation plan

Every agent evaluation eventually meets the same temptation. Early attempts come back ugly, so someone tightens the prompt; a budget stall turns into a quietly raised ceiling; a wiki page the candidates can read gets rewritten mid-run. Each edit looks small and defensible on its own, and each one corrupts the same thing: the meaning of every attempt already recorded. An attempt answers the question its plan asked, so a plan that changes while attempts accumulate manufactures evidence that looks like one experiment and is actually several, stitched together after the fact. This page is about designing plans that cannot drift — what belongs inside the frozen object, how the money divides itself, how model aliases die at plan time, and what to do when a legitimate change arrives.

Three neighbors own adjacent ground, and this page points at their work instead of restating it. The command-by-command path — linting the case, dispatching, grading, reading the report — is documented in the operator walkthrough. Setting several contestants on one identical task: the bake-off design page owns that design. What a repeated-attempt rate measures, and how little a small sample certifies, is the variance analysis. All three rest on the object this page owns: the plan that freezes the matrix before anything runs. The machinery is YYLO Benchmark's, and its lifecycle guidance states the whole posture in one line: "Planning is read-only and content-addressed." Every quotation below was checked against the committed package source, character-for-character, at evidence date 2026-08-28; every mechanical claim was read out of the package's committed code on this branch, not out of marketing.

A plan that carries its own fingerprint

The plan object closes over itself. Its identifier is a SHA-256 hash computed over the canonical serialization of every other field in the object, stored beside that content, and required to match at every later reading. Tamper with one byte — a model string, a ceiling, a wiki hash — and the identifier no longer describes the object carrying it, which is exactly the property an evaluation wants: the plan you approved and the plan that executed are provably the same bytes, or the mismatch is a hard error, not a warning. The package enforces this at both ends of the object's life. At creation, the identifier is derived from the content and the object is frozen. At every later load, the reader recomputes the hash and refuses the file if the two disagree.

Even the file write honors the semantics: a plan is written with a create-only flag, so a rerun cannot silently clobber an existing plan with a new one under the same name. A plan is an event, not a scratch file. What that event contains is the design work, and it is where most evaluations are actually won or lost.

What the hash covers, and why each family is there

The lifecycle page summarizes the resolution in one enumeration: "It resolves the exact case revision, source commit, prompt, selected wiki hashes, package and Juno versions, models, attempt count, budgets, and isolation declaration." Read as a design checklist rather than a feature list, those inputs fall into families, and each family exists because drifting there would change what an attempt means:

• The question. The case's identity — task id, ledger revision, task hash, prompt hash, and the case reference — is folded into a single input hash at lint time. After planning, the prompt is not a string in a ticket; it is these exact bytes under this exact hash, and any rewording is a different question.
• The world the candidate meets. The exported source baseline enters by its content identity; any candidate-visible wiki pages enter by path and hash; the tool policy and the typed budget object — its timeout included — enter the same way. An attempt against a different baseline or different guidance is a different attempt, and the plan refuses to pretend otherwise.
• The subjects. Exact model identities, never bare aliases, plus the record of which selector chose which identity. The next section is why.
• The repetition. The attempt count per model, fixed before any result exists.
• The money. The aggregate ceiling and the per-attempt ceiling derived from it, both hashed.
• The machinery. The benchmark package version, the YYLO version, and the isolation declaration the run will honestly claim — isolated Git objects, a trusted same-user host, no container — so the plan cannot promise an isolation level the executor does not provide.

The overview's operator sequence puts the discipline in one step: "Produce and review a deterministic plan before any canonical mutation." An abridged plan shows the shape — apart from the identifier itself, every field shown, and the ones the abbreviation omits, sits inside the hashed content:

{
  "schema_version": "juno_benchmark_plan.v1",
  "plan_id": "sha256:<fingerprint of every other field>",
  "case": { "task_id": "CASE1", "task_revision": "<ledger revision>", "input_hash": "sha256:<task, revision, prompt, and reference hashes>" },
  "models": ["<exact identity of :mini>", "<exact identity of :sol>"],
  "attempts": 5,
  "spend_limits": { "currency": "USD", "aggregate_max_usd": 20, "per_attempt_max_usd": 2 }
}

The design rule underneath the enumeration: freeze anything whose drift would change what an attempt means; leave everything else as runtime detail. The machinery errs on the side of freezing more — wiki pages, tool policy, the budget object, both versions — because an unnecessary freeze costs a re-plan while a missing freeze costs the comparison.

The ceiling divides itself before anyone can argue with it

Task-case plans bind an aggregate spend ceiling by default — twenty US dollars — and a flag selects another positive one. What makes the ceiling trustworthy is not the number but the arithmetic: the plan counts its dispatches (models times attempts), divides the aggregate across them, and floors the result at micro-dollar precision. Twenty dollars across two models with three attempts each is six dispatches at 3.333333 dollars per attempt, and the six ceilings sum to 19.999998 — the floor guarantees the parts can never exceed the whole. A clean division leaves nothing on the table: the same twenty dollars across one model with five attempts yields exactly 4.000000 per dispatch. And an aggregate too small to give every planned dispatch a nonzero share fails planning outright with an explicit refusal, rather than dispatching attempts that cannot pay for themselves.

Both ceilings live inside the hash, which converts every budget conversation into a visible, one-time event. Change the ceiling and you have not adjusted the plan; you have created a different plan with a different identifier. The live run reinforces the boundary with a typed authorization grant that must echo the plan's identity, models, currency, expiry window, and both ceilings exactly — a mismatched or expired grant is refused before dispatch. As the README states, "The grant is carried to both direct and authenticated YYLO launchers, and its worst-case reservation is retained before provider dispatch." The money is committed once, before results, in a place where second thoughts leave a fingerprint.

Aliases die at plan time

Aliases are convenient and dangerous in exactly the same way: they mean whatever the configuration currently says. The package resolves that tension at the boundary. Its README is direct: "Alias selectors such as :mini must have an exact provider/model entry in the configuration's model_aliases map." and "Planning hashes both the selector and exact identity, and execution dispatches only the exact identity so alias drift fails closed." The mechanics behind those sentences: an alias selector resolves through the project's alias configuration during planning — a bare exact identity passes through as itself — and the resolved exact identity is what enters the plan; the plan also retains which selector chose which identity, so the mapping itself is under the fingerprint. Duplicate exact models fail, and two different selectors that resolve to the same exact model fail with a refusal naming both — the same subject cannot enter the matrix twice under two names.

The payoff is temporal. An alias is a living configuration entry; a plan that recorded only the alias would silently change meaning the next time the configuration changed. Because the plan hashes both strings, it permanently answers two questions at once: what exact model ran, and what you called it on the day you asked. Re-point the alias next quarter and the old plan still names the exact identity it dispatched.

The gate between a plan and an experiment

Planning mutates nothing — no ledger write, no registry artifact — which is what makes the review step honest: the operator reads a complete object, not a proposal that already started running. Acceptance is the gated crossing, and the lifecycle page draws it in one sentence: "Review the complete plan before an authorized run creates one related canonical experiment record." The machinery around that sentence is a set of fail-closed re-checks at the moment of crossing. Acceptance re-derives the plan's hash, re-validates every model binding, and re-reads the case from the ledger immediately before any canonical mutation: a source revision that moved produces a stale-case refusal naming the expected and current revisions, and a case whose inputs no longer hash to the planned values refuses with its own error. Then, and only then, one related canonical experiment record is created, bound to the plan's identifier, with a retained receipt for the mutation.

The gate is idempotent in the right direction. If a canonical record already exists for the same plan identifier, the machinery will not mint a second experiment: it verifies the existing record binds that exact plan and, if the registry lost retained evidence, records a recovery observation and restores the linkage. Re-accepting a plan is safe precisely because acceptance itself dispatches nothing — it only writes records. Runs outside the canonical record exist only for declared fixture or local scope — an explicit, labeled escape hatch rather than a quiet second lane.

The change policy: forward in new plans, never edits

Legitimate changes arrive constantly: the case needs rewording, a wiki page improves, the package upgrades, the budget must grow, a model joins the comparison. The policy for every one of them is the same — make a new plan. The lifecycle's immutability section owns the reasoning: "New graders, reports, or investigations create derived objects linked to retained attempts rather than changing historical outcomes." Old attempts stay exactly as recorded; a later report can pool compatible evidence with provenance intact, but nothing reaches back to rewrite what ran. Adding a model is not an edit to the matrix; it is a new plan whose attempts land as a new generation beside the old one, never merged into it.

The object model reinforces the habit mechanically. A plan file is written once and never overwritten; re-planning after any change produces a different identifier, so the old and new objects sit side by side in an audit without ambiguity about which attempts belong to which question. That is also why hand-maintained version labels on evaluation documents decay: the identifier is the version, computed rather than asserted. When someone later asks what exactly ran in March, the answer is a hash they can re-derive from the retained plan bytes — or discover they cannot, which is itself a finding.

What a mutable plan would do to the results

The statistics of repeated attempts are the variance analysis's subject; the pooling rule that decides which attempts may be compared is spelled out on the bake-off design page. This page's piece of the argument is narrower and prior: each attempt is a draw from a distribution defined by the plan's frozen inputs, so comparability is manufactured at planning time and can only be spent, never recovered, afterward. Edit the plan mid-series and the ledger still counts one experiment while the attempts answer different questions. The classic poisoning modes are all variations on this: a prompt tightened after early failures makes the early failures measure a question nobody is asking anymore; a model added after the standings are visible arrives with knowledge the first contestants never had; a ceiling raised after a stall mixes two cost regimes into one cost column; an alias re-pointed mid-series lets one experiment label cover two different models.

A mutable plan makes each of these invisible. A content-addressed plan makes each of them impossible to hide: the experiment record binds the plan identifier, every reader re-derives it from the bytes, and a mismatch between what was approved and what ran is a checkable predicate instead of a matter of whose memory to trust. Immutability here is not ceremony — it is what separates an evaluation from a story that had a budget.

Ownership and neighbors

This page owns one job: how to design the frozen plan object — what belongs in the hash, how the ceiling divides, how aliases resolve, and the forward-only policy for change. Running the whole loop command by command is the operator walkthrough's lane; comparing several contestants under one matrix is covered by the bake-off design page; what repeated attempts measure is the variance analysis; and which evaluation instrument answers which kind of question is the instrument taxonomy. Two deeper companions — honest cost and failure-evidence semantics, plus a full methodology reference for the package — are approved but not yet published in this program. The next-step panel is the single primary action: install the evaluation package, and freeze your next comparison with it.