Regrade benchmark attempts under new criteria, without rerunning candidates

Sometime after the first experiment, the grader gets better. A rubric that was too loose turns out to accept work the tests would reject; a checker misses a failure mode nobody had named when the case was authored; the criteria simply mature. Now every attempt already graded sits under verdicts you no longer fully believe, and two expensive instincts compete. The first is to re-run the whole experiment — pay every model again, rebuild every patch, and hope the new verdicts land on identical work. The second is to edit the old results in place, quietly replacing the numbers history saw. The first spends money to learn nothing the archive does not already hold; the second destroys the only record that the criteria ever changed. The operation this page documents refuses both: it applies governed grading to the attempts you already retain, in their byte-addressable form, and records what the new criteria say as fresh evidence stacked on top of the old — never in place of it.

The regrade path belongs to YYLO Benchmark's task-case experiments — plans that expand a Kanban case into bounded, immutable attempts — as the package stands on this branch. Quotations below were matched against the committed package bytes character for character, and every mechanism described here was re-read from this branch's source before the page was written (evidence date 2026-08-29). Each neighbor keeps its own territory and is linked below instead of being retold: the walkthrough owns the operator loop from case authoring to report; the governed-grading page owns verdict semantics and the receipt chain; the frozen-plan guide owns the immutable object attempts hang from; the salvage page owns interrupted runs; and the variance guide owns the reading of attempt rates.

What a regrade may touch, and what it refuses

An attempt that finished running is not a memory of work — it is a small archive of artifacts: the contract that fixed its identity, the captured output streams, the rebuilt diff, the structured terminal evidence, the normalized candidate result, and the verdict's sealed receipt. A regrade is permitted to read exactly the inputs that determine a verdict, and nothing that could produce new work. The boundary is the package's own one-line contract: "Re-run governed grading from retained candidate artifacts; no candidate runner or workspace is accepted." The README states the reach in the same breath: "The public regradeExperiment API consumes retained attempt/candidate/patch evidence and can append a new grading generation without accepting or rerunning a candidate runner."

Mechanically, the operation walks the experiment's verified manifest and picks out three artifact roles — the attempt contracts, the candidate results, and the structured terminal evidence — and regrades attempt by attempt, demanding each attempt's candidate result before that attempt's verdict can move. An attempt whose candidate evidence was never retained is not regradeable, and the refusal names it: the operation stops with "candidate evidence missing for" the attempt's identity rather than grading a hole or skipping onward. The walk appends durably as it goes, so a refusal that fires partway through leaves the earlier attempts already regraded — which is why the real precondition is an experiment whose evidence is complete. Finish or salvage the run first (the salvage page owns that ladder); a regrade is a verdict operation, not a gap-filler.

The no-rerun property is tested, not merely asserted. The package's release gate refuses to pass without an executed positive control whose case id is regrade:no-candidate-rerun, and the control only counts when its observed facts prove the candidate call count identical before and after a regrade with exactly one regraded result — a real run, then a real regrade under a versioned grader, with the candidate counter provably unmoved. A regression that quietly re-dispatched candidates during regrading could not ship.

The flow: new grader bytes, one unchanged plan

The criteria change never touches the plan. A plan freezes the *name* of the grading profile — one field of the case reference — while the profile itself lives in your project configuration, where it binds an executable path, a SHA-256 over that executable's bytes, a grader ID, and a grader version. To mature the criteria, you write the better grader, bind it under the same profile name with a new digest and a new version string, and leave the plan file exactly as it was. The regrade re-validates that plan file on the way in — its content hash is re-derived, and a plan whose bytes drifted is refused — before a single artifact is read.

Then the command is one line, yylo-benchmark regrade --plan checkout-plan.json, with YYLO_BENCHMARK_REGISTRY naming the private registry, because every command that retains or reads evidence declines to operate without a registry root. The command's own description carries the whole posture: "Regrade retained candidate evidence without candidate execution." When it finishes, it prints the regraded results as canonical JSON, and the canonical experiment record — the one related record the plan's identity resolves to — is rewritten to terminal status with the newest per-attempt summaries, the number resolved, and the evidence digests, its mutation receipt appended onto the experiment. A plan whose canonical record has vanished gets nothing minted in its place: the operation stops with "canonical experiment record is missing for regrade", because a regrade grades history; it does not invent it.

The success fact is re-read, never re-asked

Nothing about the candidate is asked twice. The candidate-success flag that grading consumes is re-derived from the retained terminal evidence — the recorded envelope's status together with the recorded exit code — not from the grader, not from any live process, and not from the candidate's own account of itself. The verdict conjunction then holds exactly as it held at run time: an attempt counts as resolved only when the retained evidence says the candidate succeeded and the new grader's boolean comes back true. A candidate that failed on its own merits keeps the failure it earned, no matter how generous the new rubric is; a clean run the new rubric rejects becomes a grader-failure terminal — typed, visible, and final. And a regrade launched with the profile unconfigured does not crash — it appends fail-closed generations under a missing:<profile> grader identity with the verdict recorded false, the recorded-failure discipline the governed-grading page documents. Configure the profile before regrading, or you will stack honest-but-useless missing-verdict generations on top of your real ones — and they become current.

One boundary deserves precision. The identity agreement a live run checked — observed provider and model against the planned identity — is retained inside the terminal evidence, but the regrade's success flag derives from the envelope status and exit code alone and does not re-evaluate that agreement. For a normal experiment this changes nothing, because the retained evidence was already reconciled when it was written. But an operator regrading an experiment that contains identity-mismatched attempts should read those attempts' retained evidence first; under a passing new grader, the regrade's verdict for such an attempt follows its evidence fields, not the harness-failure class the live run assigned. The bytes tell you which situation you are in before you commit a generation.

Generations stack, and sequence decides currency

The registry is an append-only log. Every artifact lands with a sequence number and — except for the first — a hash of the manifest entry before it, and nothing already written is ever modified — a new grading generation is new bytes at a new address, appended after everything that came before. Currency is therefore a rule about order, not a pointer anyone maintains: when reports and audits collect results, they walk the manifest in sequence, and the last result recorded for an attempt is the one that stands. Older generations do not vanish; they stay addressable at their own digests, and the report layer retains every grader result ever recorded for an attempt, so a version-one verdict and its version-two successor are both visible to anyone reading the experiment.

Two consequences follow. First, re-running the same regrade with a deterministic grader over unchanged evidence reproduces byte-identical verdict artifacts, and the registry's content-deduplicated appends notice: no new verdict bytes land, because those bytes already have addresses — only the canonical-index rewrite repeats, appending a fresh ledger mutation receipt each time. Second, the standing audit keeps pace by itself — the doctor command rebuilds the current-verdict view the same way, last in sequence wins, and walks the grader-receipt chain again for every attempt that currently stands resolved or in grader failure, reading the newest receipt first. A regraded experiment does not need a new audit ceremony; it needs the same one, run again.

Which grader graded what

Every generation carries its author. A grading generation's typed result records the grader's ID and version beside the verdict it produced, and the same pair is sealed into the integrity-hashed receipt next to it. Two generations of one attempt read as two schema-exact facts:

[
  { "schema_version": "juno_benchmark_grader_result.v1", "attempt_id": "<derived attempt id>", "grader_id": "suite-checker", "grader_version": "1", "required": true, "passed": true, "evidence_hash": "sha256:<digest of the grader's output>" },
  { "schema_version": "juno_benchmark_grader_result.v1", "attempt_id": "<derived attempt id>", "grader_id": "suite-checker", "grader_version": "2", "required": true, "passed": false, "evidence_hash": "sha256:<digest of the grader's output>" }
]

A stricter version two rejecting what version one accepted is exactly the story the pair above tells — and the disagreement is retained, not reconciled away. Which layer a report reads is explicit. Name a grader version when generating a report, and it includes only the attempts that grader version graded; the default layer, normalized-result.v1, reads every retained attempt's current verdict. The grader version is also hashed into every cohort key alongside the case, snapshot, and prompt inputs of the frozen compatibility contract that the variance guide and the walkthrough enumerate — a key always carries the grading layer it was computed under, so cohort identities from differently pinned reports can never be silently equated. Inside the default layer the folded value is the layer's own name, not each attempt's grading generation; that layer reads current verdicts by definition, so pin the report to a grader version when the separation must be explicit. And reports are frozen artifacts themselves: "Retain each report version as a new derived manifest layer; source experiment manifests are never changed." The report you circulated before the regrade still exists, byte for byte, beside whatever you generate after it.

Comparability rules for regraded evidence

The mechanics above make regrading safe; these rules make it honest.

Label every published number with the criteria that produced it. A rate graded under version two and a rate graded under version one are two different measurements that happen to share a case. Publishing them side by side is fine and often the point; publishing the newer one as though the older never existed is exactly the history edit the append-only design exists to prevent.

Compare stored reports, not regenerated ones. The pre-regrade report artifact is immutable; regenerate after a regrade and the default layer will read the new current verdicts, so the difference between the two stored reports *is* the regrade's measured effect. Present both, dated, and let a reader see the flip — the variance guide owns what such a pair of rates can and cannot certify statistically.

Regrade to improve criteria, never to shop for verdicts. A regrade run to rescue a disappointing number still works mechanically — and still leaves the older generation sitting beside the new one, both receipted, both auditable by anyone who runs the doctor. If the new verdict is a better measurement, say why the criteria improved and show both generations; a flip that cannot survive that explanation was not a measurement.

Regrading is not recovery. When a crash left an attempt with candidate truth but no verdict at all, that sequencing belongs to the salvage page, whose rerun-then-regrade ladder this page deliberately does not restate; regrading is for verdicts you already hold and no longer endorse. And it is not re-planning either: if the case, the prompt, or the attempt matrix itself needs to change, that is a new plan and a new experiment — the frozen-plan guide owns why.

Task-case regrade, workflow rejudge

Tracked-workflow experiments — governed agent steps rather than Kanban-case attempts — have their own re-grading operation, rejudge, with its own hash-chained judgement generations, and the CLI refuses to blur the two worlds: rejudge pointed at a task-case plan stops with "rejudge supports workflow plans only; use regrade for task-case plans", and the regrade command conversely reads task-case execution plans only. The rejudge machinery — blinded candidate truth, generation chains that must increment by exactly one, durable intent before every judge call — is the governed-grading page's territory and stays linked, not restated. The split is not bureaucratic: a task-case regrade re-applies an executable grader to retained patches, while a rejudge re-runs a governed model judge over retained blinded truth, and the two evidence chains are shaped differently all the way down.

Where this page sits

Everything here serves the regrade alone: what it may read, what it refuses, how generations stack, and how to compare what they produce. Around it, the walkthrough runs the loop that creates the evidence a regrade consumes, the governed-grading page owns the verdict machinery it re-applies, the frozen-plan guide owns the plan the profile name lives in, the salvage page owns the interrupted-run ladder that sometimes ends here, and the operational command reference is the benchmark documentation. When your rubric outgrows its first draft, the machinery above lets the criteria move without the bill moving with it; the panel below installs it.