Blinded and governed grading for coding-agent output

Every agent evaluation funnels into one uncomfortable minute. The attempts have run, the diffs are sitting in their checkouts, and now somebody has to say whether each one passes. Almost every way an evaluation lies starts right there: the judge recognizes the contestant and reads its work charitably; the judge was quietly edited after the early results came back; the rubric drifted between the first candidate and the last; the verdict arrives as a paragraph of hedged prose instead of a decision; the record of what was judged, and by what, is kept by the same hand that had opinions. None of these are character flaws — they are structural properties of an ungoverned grading step, and structure can be fixed mechanically. This page is the fix as one protocol: pin the judge, control what it can see, force a binary verdict, govern disagreement, and seal everything into records a stranger can check.

The machinery used throughout is YYLO Benchmark's, but the protocol is written to travel. Four neighbors hold the adjacent ground; this page defers to each rather than restating any: the operator walkthrough owns the command-by-command loop; the bake-off page owns the several-contestant design; the immutable plan guide owns the frozen plan object everything else presupposes; and the variance treatment owns what a repeated-attempt rate can honestly certify. Quotations below were copied from the committed package text and re-checked verbatim on 2026-08-28; every mechanical statement was checked against the package's committed source on this branch, behavior re-derived from the code rather than recalled from prose.

The five ways a judge gets bent

Name the failure modes first, because the protocol is only as good as the list it answers. Five channels cover most grading bias in agent evaluation:

• Identity leakage. The judge knows which agent, model, or team produced the work, and that knowledge moves the verdict — leniency for a favorite, scrutiny for an underdog. The cure is not a promise of impartiality; it is making the channel physically absent from what the judge receives.
• Post-hoc mutability. The grader is a script someone can edit, so it is quietly amended once the disappointing results arrive — and history silently re-scores itself. The cure is binding the judge to exact bytes, checked at every use.
• Rubric drift. For a model judge, the prompt or rubric that defines what counts as good is edited between candidates, so the two were never asked the same question. The cure is hashing the rubric into the judge's identity before any result exists.
• Verdict vagueness. A verdict of mostly-works-with-caveats is not a verdict; it is an escape hatch that converts later argument into apparent success. The cure is a typed boolean plus retained written evidence — nothing else crosses the boundary.
• Erased disagreement. When a second grading disagrees with the first, someone replaces the old record with the new one, and the disagreement — which is information about the grader — disappears. The cure is append-only generations with their own integrity.

Each of the five gets its mechanism below, in the order a grading step encounters them.

Bind the judge to bytes before it can judge

Start with the executable path, where the judge is a program. The configuration refuses an anonymous judge: "Each case's grader_profile must select a configured grader_profiles entry bound to an executable SHA-256, grader ID, and version." A profile names the exact file, the exact identity strings, and the exact digest of its bytes — so a grader is not whatever the check script happens to be today, but one immutable object with a name and a checksum.

The runtime enforces that binding at both edges of every call. The grader file must be an ordinary regular file — a symlink is refused on sight; its bytes are hashed and compared with the configured digest before it runs, and hashed again after it has answered, so a grader that changed mid-grading is caught rather than trusted. It is spawned directly, never through a shell, with nothing inherited from the operator's environment except a path to find executables in — no stray credentials, no HOME, no variables that could alter its behavior between two candidates. Its input arrives as one canonical JSON document on stdin, and a default two-minute ceiling ends a hung judge with a hard kill rather than an evaluation that waits forever. When it returns, three things must hold: the process exited zero, the output parsed as JSON, and that JSON carries one boolean passed. The pipeline reads exactly one field of that output — the boolean — while the whole output object is retained as the grader's evidence artifact; any reasoning a judge appends can explain but never decide. And a judge whose identity strings arrive blank is treated like a missing one, its grading failing closed into the recorded-failure path rather than into a verdict.

What the judge sees — and what "blind" honestly means

Here the protocol splits by judge type, and precision matters more than slogan. When the judge is an executable in the task-case path, its stdin record carries five families of fact: the attempt contract under its hashes, the grading profile's name, a flag for whether the candidate run itself succeeded, the patch digest, and the terminal-evidence digest. That input is not anonymous — it honestly includes the attempt's identity. What this path guarantees is verdict-path blindness: the judge is fixed, hashed, byte-pinned machinery, so no amount of knowledge about who produced the work can change what the judge is. A mechanical judge cannot be charmed; it can only be edited, and editing is what the byte-binding prevents.

When the judge is a model rather than a program, input-level blinding becomes possible and the workflow path enforces it structurally. The governed judge receives exactly three things: its own governed identity, the scoring id naming what is being scored, and the blinded candidate. The blinded candidate is a canonical JSON document holding the schema label, the scoring id, the hash of the retained candidate truth, a transcript, and the artifacts — nothing else. Which model produced the work, on which provider, under which session ids, at what cost: none of it is in the judged payload, because none of it is in the object. All of it still exists — it is retained in the evidence receipt beside the judgement — but it never crosses into the judge's field of view. The judge knows what it is judging; it cannot know who it is judging.

The transcript and artifacts get redacted before they reach that point, and the redaction is itself governed. Secret patterns configured in the policy scrub the transcript and every artifact; the machinery then re-tests the retained text against every pattern and refuses to proceed if a single match survives, so an incomplete scrub is a hard stop, not a warning. The receipt records how many patterns ran, how many replacements were made, and stamps the result clean — redaction evidence you can point at, not a promise.

The judge's own identity is governed the same way the executable's is, by hashing instead of hope. The policy pins the judge's id, its version, the exact model identity it runs on, and the hash of its rubric; every judgement then carries a judge-policy hash computed over that object, so a judgement names the exact judge policy that produced it, and a rubric edited between two candidates is a different judge by construction.

The verdict is a boolean, and failure fails closed

Nothing gets called resolved on the judge's word alone. Two independent facts must both hold: the candidate process itself succeeded, and the grader's boolean came back true. A candidate whose process finished cleanly but whose judge says no lands in the grader-failure class — a typed outcome, never a rescue, never a quiet pass. A candidate that failed retains the failure it earned; no lenient judge can upgrade it. The workflow path adds a third input to the conjunction: a candidate whose harness validity was marked invalid can never be judged resolved, and a judge refusing an otherwise-successful candidate produces a distinct judge-failure terminal class. The eligibility facts come from the candidate side, the boolean from the judge side, and the machinery intersects them — which is why the package's README can state the whole dependency as an artifact chain: "Required grader input, output, result, and integrity-linked receipt artifacts determine resolution."

Failure of the grading step itself produces evidence rather than a gap. A grader that is missing, unconfigured, crashes, times out, or comes back with a malformed answer gets its failure stamped into the record — identity recorded as missing:<profile>, version as unavailable, verdict false — and the full input, output, result, and receipt artifacts are still written and still chained. A missing judge is therefore an auditable event with a paper trail, not a silent hole in the data. The same posture guards the model-judge boundary: judge calls travel through the same hash-pinned reviewed module as candidate dispatches, and there, "malformed, timed-out, oversized, identity-mismatched, or nonzero responses fail closed". The module's diagnostics stay inside the boundary — its error output is never reflected into public errors or receipts — so a judge failure surfaces as a typed refusal, not as leaked operator state.

When graders disagree, append a generation

Graders improve, rubrics get sharper, and a second grading will sometimes disagree with the first. The protocol's answer is that disagreement is data about the grader, and data is appended, never overwritten. In the workflow path every judgement carries a generation number: the original run produces generation one, and each governed rejudge increments the prior generation by exactly one, with each new judgement chained to the previous one by its hash. The chain is verified, not assumed — a rejudge receipt that cannot name the judgement it follows, or that skips a generation, is rejected as invalid history.

The rejudge path is fenced on every side. The README describes the ordering discipline: "Rejudge writes a durable identity-bound intent before the governed call, without financial authorization." — the intent to judge is persisted before the judge is ever invoked, so a crash mid-judgement leaves durable evidence of the attempted call rather than an invisible side effect; and if a prior judge dispatch's effect cannot be proven, the machinery demands manual recovery instead of guessing. What the rejudge consumes is fixed: "Rejudge reads the complete content-addressed receipt set, dispatches no candidate, and appends a new governed judgement generation plus report." The candidate is never re-run, never consulted, and cannot even be re-dispatched through this path — the lifecycle page closes the loop in one line: "Rejudge persists identity-bound intent before judge dispatch, consumes retained blinded candidate truth, and cannot rerun a candidate."

The task-case path holds the same line through its grading generations: a regrade consumes only retained candidate evidence and appends new graded results, and the reporting layer keeps generations from melting together by folding the grader version into every cohort key, while a report pinned to a given grader version admits only attempts that version actually graded — a regraded generation is read as its own report layer beside the original, never averaged into a differently pinned one. The workflow report likewise reads the tip of each judgement chain as the effective verdict while the superseded generations stay addressable beside it. A disagreement between generations therefore remains visible for exactly as long as the retained evidence does.

Governance records a stranger can audit

The last layer is what makes the rest credible to someone who was not in the room. Every grading appends a receipt that seals itself:

{
  "schema_version": "juno_benchmark_grader_receipt.v1",
  "attempt_id": "<hash-derived attempt identity>",
  "grader_profile": "<profile name from the plan>",
  "grader_id": "<pinned grader identity>",
  "grader_version": "<pinned grader version>",
  "required": true,
  "passed": false,
  "input_hash": "sha256:<the exact stdin bytes>",
  "output_hash": "sha256:<the exact grader output>",
  "result_hash": "sha256:<the typed graded result>",
  "integrity_hash": "sha256:<hash of every field above>"
}

The integrity hash is computed over every other field in the receipt, so editing any of them — flipping passed, swapping the grader version, relabeling the profile — breaks the seal visibly. Verification re-derives that hash, then goes further: the three referenced artifacts must exist in the experiment's manifest and read back as bytes, and the graded result's own fields must agree with the receipt's — attempt, verdict, grader id, grader version, all four checked. The doctor command re-runs exactly this check for every attempt whose final state is resolved or a grader failure, which makes receipt verification a standing audit rather than a one-time ceremony. The workflow path seals the same way at one level up: an evidence receipt whose checksum covers the observed identity, both session layers, runtime, cost, the redaction tally, and the judgement together, plus a candidate-truth object that binds the judged transcript and artifacts by reference and hash. And the trust rule for rejudging is the auditor's rule: the digest a rejudge must match comes from the immutable ledger, never from the receipt a caller hands over — the operator cannot grade against a receipt they just forged.

Put together, a third party with no trust in the operator can recompute every checksum, re-read every referenced artifact, re-derive the candidate-truth binding, walk the judgement chain, and confirm that the verdict published is the verdict the bytes contain. That is the difference between governed grading and grading with good intentions.

The protocol, compressed

• Pin the judge before any result exists: exact bytes by digest, exact identity strings, exact version — checked at every use, before and after.
• Type and narrow the inputs: one canonical document in, one boolean plus written evidence out; no shell, no inherited environment, no unbounded wait.
• Strip identity structurally when the judge is a model: the judged payload contains the work, not the author; redaction is verified, counted, and receipted.
• Hash the rubric into the judge's identity, so an edited rubric is a different judge with its own generations.
• Make resolution an intersection: candidate success and grader assent must both hold; refusal and judge failure are typed outcomes, never rescues.
• Let grading failure produce records: a missing or broken judge is retained evidence, not a hole.
• Handle disagreement by appending: generations chain by hash, reports read the tip, history stays addressable, and cohorts never merge across grader versions.
• Seal everything into receipts whose checksums cover their own fields, and force rejudging to trust the immutable ledger, not the operator's copy.

Where this page sits

The grading method is this page's charge: the blinding protocol, the disagreement governance, and the audit records. Around it, the operator walkthrough runs the loop, the bake-off page fields several contestants on one task, the immutable plan guide freezes the matrix, and the variance treatment reads the rates; the package's own pages under the benchmark docs carry the operational reference. Four subjects touched in passing — recovering an interrupted run, the honest semantics of cost and failure evidence, the regrade operation as an operator flow, and the benchmark package's methodology reference — are each approved as a later page in this program. When this protocol grades your next evaluation, reference it by the canonical URL in the panel underneath and let the receipts do the arguing.