Benchmark coding agents on a real repository

Published availability first

The thin runner is published: Benchmark 0.2.0 introduced this lifecycle; Benchmark 0.2.1 adds frozen project/task criteria and deterministic loss. CLI 0.2.11 requires Benchmark 0.2.1 exactly and Skills ^2.1.1. The v3 identifier denotes the record schema, not a package version. Registry publication and exact archive documentation were reviewed on 2026-10-07; installing a package does not upgrade your existing runtimes or skills automatically. See the version changelog, implementation study and evidence limits and version-labeled documentation. Verify installed help before using examples. For checklist judgments, read the frozen-criteria contract and canonical Benchmark skill.

Review the case before dispatch

Choose one meaningful repository change and behavior-based checks. For historical work, reconstruct original requirements and the complete pre-solution range; do not assume the completion commit's parent is the base. A read-only Ledger proposal omits the completion response but still needs review:

yylo-benchmark case draft --ledger-task TASK_ID > /tmp/case-draft.json

Write the reviewed requirements to a prompt file. Case storage must be outside the source repository. Replace the path and commit placeholders with reviewed values:

yylo-benchmark case create \
  --source /path/to/repository --base PRE_SOLUTION_COMMIT \
  --reference SOLUTION_COMMIT --prompt /tmp/requirements.md \
  --exclude private-answers --output /tmp/cases/my-case --reviewed

Reference provenance is optional and is not copied into candidate input. Review the baseline/reference controls and exclude completion records, hidden checks, answer documents and other attempts. Prepared snapshots omit future Git history; default exclusions include .juno_task, .gitmodules, hidden-graders and reference-solutions. Symlinks and gitlinks must be excluded or materialized in a separately reviewed input repository. A library case still requires this review; it is not a drop-in legacy task schema.

Declare and run independent treatments

Save this treatment as /tmp/model-a.json, replacing the model selector with one your installed harness supports:

{
  "name": "pi-model-a",
  "model": "provider/model-a",
  "harness": "yylo_pi",
  "executable": "yy",
  "args": ["--thinking", "medium"],
  "timeout_ms": 1800000,
  "configuration": {}
}

yylo-benchmark run --case /tmp/cases/my-case \
  --treatment /tmp/model-a.json --attempts 1 \
  --output /tmp/experiments/comparison-1

The output directory must be new. Each attempt has an independent fresh starting repository. Harness setup owns dependencies and supported local initialization; Benchmark does not borrow dependencies, repair workflows or silently retry. Live model calls require your explicit authority and budget. A one-shot result is not a reliability estimate; changing harness/configuration prevents model-only attribution.

For workflows, capture a tracked YAML file with case create --workflow path/in/source.yaml. Use a workflow_runner treatment with the installed runner path, a declared model variable and optional workflow.through step ID, as shown in the source README. Each variant runs its own prefix from the same initial input through that step, then stops. Results include upstream effects, not just the selected step. There is no downstream continuation. The workflow owns sessions, dependencies and errors; universal cross-harness compatibility is not promised.

Evaluate retained output without rerunning candidates

A check command receives a JSON packet on stdin and returns a JSON assessment. Save an evaluator specification as /tmp/checks.json:

{
  "name": "behavior-checks",
  "kind": "check",
  "command": {"executable": "python3", "args": ["/private/checks.py"]},
  "timeout_ms": 600000
}

The private checker must return an assessment such as {"verdict":"fail","findings":["Missing required behavior"]} and exit zero. Nonzero exit or malformed output is an evaluator error, not a candidate test failure.

yylo-benchmark evaluate --attempt /tmp/experiments/comparison-1/1-1 \
  --evaluator /tmp/checks.json
yylo-benchmark report --root /tmp/experiments/comparison-1 --table

Later, supply a new check, judge or human profile to evaluate. No original-catalog prebinding is required, candidates are not rerun, and each evaluation retains its own specification and result. Prior assessments remain intact. Execution, check failures, judge disagreement, evaluator errors and unknown costs remain separate; no automatic winner or combined verdict is calculated.

Preserve incomplete or disqualified evidence

An unfinished intent appears as interrupted_or_running. Inspect its producer and evidence; never infer safe replay. There is no automatic retry or resume. An authorized new experiment uses a new output directory. report can show partial results without presenting them as complete.

yylo-benchmark disqualify --attempt /tmp/experiments/comparison-1/1-1 \
  --reason 'Confirmed reference solution exposure'

This appends a note, not a replacement result. Keep old evidence and its original implementation; legacy plans are not automatically migrated.

Boundaries that belong in every report

This is trusted-host workspace/context hygiene, not enforced filesystem/account/network isolation. Host authentication is inherited, and candidates may access other host paths or public answers. Known answer exposure disqualifies an attempt; matching a reference alone is not proof of cheating. Checksums detect drift, not a malicious host rewriting everything.

Retained prompt input is not proof of literal provider-message delivery: harness preprocessing may change it. Requested model identity and observed identity are separate. Missing cost is unknown, not zero or an invoice. Keep credentials out of prompts/configuration and retained logs. The fifteen-task study reports retained implementations, revised checklist assessments, unequal time budgets and incomplete costs; it is not a universal model ranking.