Ten CLI tasks, forty attempts: what our benchmark actually taught us

Why use our own historical tasks?

A coding agent can produce a plausible patch without delivering the behavior a repository needs. We wanted to examine that gap on real CLI work: command forwarding, prompt references, lifecycle ownership, dependency hydration, runtime readiness, and safe retirement of execution paths.

We selected ten historical tasks from our CLI's public source. This was a purposive case study, not a random sample of software work. Each case had a reconstructed starting point and a retained reference implementation. The question was narrow: could a one-shot attempt satisfy the selected task's focused checks and substantive review?

The reference was a control, not an infallible answer key. A check that distinguishes the baseline from the reference can still reject a valid alternative implementation. That distinction became one of the experiment's central lessons.

The experiment in brief

• 40 scored candidate attempts: ten cases, four models per case, one attempt each. Cases 1–2 retained their earlier results and evaluation repairs; the continuation added cases 3–10, or 32 attempts.
• Candidate selectors: openai-codex/gpt-5.6-luna, openai-codex/gpt-5.6-sol, openai-codex/gpt-5.6-terra, and openai-codex/gpt-5.5. The table abbreviates the first three to Luna, Sol, and Terra. These are the recorded selectors, not a claim about general availability.
• Judge: openai-codex/gpt-6-astra. Candidates and judges used yy pi with medium reasoning.
• One-shot policy: no candidate was replayed to improve a score. Two attempts reached the 45-minute harness budget.
• Controls before dispatch: every case passed baseline/reference controls. The historical implementation range included final repair commits rather than assuming an integration commit's immediate parent was the correct starting point.
• Reproduction environment: fresh Git histories without future commits, Node 22, task-local exact-lock installs, and same-base build twins where needed. Installed YYLO 0.2.8 initialized Simple replicas; this was not a complete reenactment of each historical host runtime.

These were trusted-host executions, not a filesystem/account sandbox. Actual sessions remained in shared account storage. Read restrictions were policy, not proof of isolation. No shared launcher was activated, and no product deployment occurred during the continuation.

What counted as success?

Acceptance required both the applicable focused checks and substantive review, subject to the study's limitations. Green tests alone were not enough. Conversely, an incomplete measurement package was not automatically a demonstrated code defect.

The final report records 2 accepted tasks each for Luna, Sol, and Terra, and 1 for GPT-5.5. Retained patches passed every focused check on 4, 6, 4, and 3 tasks respectively. Those are different measures: GPT-5.5's task 3 patch passed its checks even though the attempt timed out.

Here is the complete per-task picture, including the separately corrected evaluations for cases 8–10. Each cell gives passed/total focused checks and an outcome. P means accepted; N means not accepted with implementation or behavioral concerns; V means focused checks passed but required validation or measurement delivery was incomplete; T means timeout with overall delivery quality unknown; ? means conservative evaluation uncertainty after correcting the oracle.

Cases 7–8 count end-to-end scenarios, not individual assertions. Do not average these fractions into a cross-task score.

Green checks and complete delivery are different

The wiki proxy was the cleanest result: all four attempts passed 30/30 checks and review after previously documented grader repairs. The execution-status unit fixture also passed for Luna, Sol, and Terra. GPT-5.5's retained fixture patch passed 13/13, but its attempt timed out; a good-looking patch did not create a successful terminal record.

The prompt-reference case showed why test counts cannot stand in for acceptance. Sol passed 38/38 frozen checks, yet two additional malformed-boundary bugs were independently reproduced. The original denominator stayed 38 rather than being silently expanded after the result.

The schema-cache task showed the opposite risk. All four passed 6/6, and review found the operation-local cache implementation sound. What was missing was the required paired fixture measurements and wider phase/count evidence. Calling that four broken cache implementations would misrepresent the finding. Sol's lifecycle-ownership patch likewise passed 42/42 without an established runtime defect in review, but omitted required nested-delegation, interruption, and cleanup validation.

Some failures were concrete. All four watch implementations timed out while following a dead producer without terminal evidence. All retained task-retirement implementations left an exported direct-invocation route unguarded. The launcher-activation case exposed completeness or authentication problems, and every readiness attempt failed the packed public-wrapper scenario. These observations are narrower, and more useful, than saying an entire model is good or bad at engineering.

We had to correct our own answer key

Historical tests sometimes encoded the reference implementation's wording or private helpers rather than the behavior the task actually required. That was a benchmark methodology defect, not evidence of candidate incapability.

For scoped hydration, an exact error-message assertion rejected valid stale-lock refusal. Correcting only that assertion changed Luna, Sol, and Terra from 0/1 to 1/1; GPT-5.5 remained at 0/1 because its selection evidence disagreed with later verification. A separate simple legacy fixture passed for baseline, reference, and all four candidates. It did not corroborate the broader legacy-regression concern in the original reviews, but it also did not prove every configuration safe. Without replacement review, the three corrected passes remain inconclusive, not accepted.

For watch, admitting semantic retirement wording changed the counts from 29/35, 32/35, 29/35, and 31/35 to 30/35, 33/35, 29/35, and 32/35. The dead-producer failure remained.

For task retirement, removing a reference-private helper test and correcting rigid wording changed the denominator from 51 to 50. Counts changed from 43, 44, 40, and 44 passes to 44, 46, 43, and 45. The corrected suite does not claim replacement historical-status coverage; it still requires direct runtime safeguards.

The correction protocols were frozen before deterministic-only evaluation. Original results remain immutable. No extra candidate executions or judge calls were used for these corrections. An improved oracle should not erase its own history.

The most important audit checked what the models actually received

Freezing a prompt file proves what we authored. It does not prove what arrived in the model's session.

An audit of the first user message in all 40 candidate sessions found that installed Record hydration changed every authored prompt. Markdown headings such as ## Benchmark became timeout-warning blocks instructing manual Record lookup. Removing those warning substitutions reconstructed the original text apart from blank lines.

There was one identical delivered payload per task across its four models, and no hydrated Record payload marker was found. That makes the deviation common injected noise and instruction conflict, not literal delivery of the frozen prompt. Equal noise does not make the experiment pristine.

A limited tool-argument audit identified no manual Record lookup in response to the warnings. Two string-pattern hits were source-writing heredocs containing example commands, not executed lookups. This is not proof of no contamination. The setup canary and availability smoke had not tested the actual transformation. Future runs need to verify the delivered message before scoring, not just the launcher input.

Integrity checks still failed

All 32 new candidate manifests independently verified unchanged after grading and corrections. That is useful evidence of retained-byte stability, but it is not a native doctor/report pass.

All eight new native reports failed: cases 3–5 and 7–9 rejected their own log references; case 6 flagged credential-like bytes in a historical document; case 10 rejected a skipped-judge profile-hash linkage. The flagged case 6 document was byte-identical to baseline for every candidate, which does not certify it secret-free. No scanner was suppressed and those bytes are not reproduced here.

These failures limit what we can claim about the complete evidence pipeline. An independent manifest check cannot stand in for the native chain of plan, execution, evaluation, and report verification. Nor do focused source-level replays certify original production deliveries, full suites, release gates, or live timing targets.

What did it cost, and how long did it take?

Known reported estimates across retained setup, smoke, candidate, and judge entries totaled USD 56.06619488. That is a rate-card estimate with incomplete coverage, not an invoice. Fifteen earlier fully unknown entries remain unknown. Two native timeout costs remain unavailable; separately bound partial-session estimates are included in the known total, without converting the native unknowns into complete billing records.

Candidate-only totals were:

• Luna: 1h 29m 35s; USD 0.6560 known estimated cost.
• Sol: 2h 24m 40s; USD 20.9918 known estimated cost.
• Terra: 0h 58m 40s; USD 4.2606 known estimated cost.
• GPT-5.5: 1h 52m 11s; USD 21.2909 known estimated cost.

Candidate wall time excludes launcher hydration and grading/judging, but includes CLI preprocessing and candidate-invoked tests. Timeout intervals are estimated from dispatch start to the native timeout terminal, not completed-run durations. Sol spent about 42 minutes on task 3, mostly on broader validation; that is not model-token latency.

Fixed model order, host load, and different task 1 evaluation cohorts further limit comparisons. Assistant, host-compute, and human effort are excluded; actual subscription charges are unknown. These numbers do not support a cheapest-model or fastest-model recommendation.

What we would require before a publishable comparison

This experiment exposed both candidate defects and defects in the measurement system. Keeping those categories separate is the result worth carrying forward.

Before a defensible comparison, we need repaired native integrity/linkage, verified literal prompt transport, reviewed implementation-independent oracles, and resolution of the three scoped-hydration uncertainties. Replication needs separate authorization; this one-shot study cannot establish repeated-run reliability or general success probability.

For teams designing their own evaluations, the practical lesson is to test the benchmark itself: verify the actual session payload, prove the baseline/reference controls discriminate intended behavior, retain unsuccessful attempts, and keep unknowns visible. A useful report should make it easy to see where the evaluator was wrong, not merely where the candidate failed.

For current usage, see the YYLO Benchmark documentation and source repository. This retrospective describes the retained experiment, not a guarantee about the current package's behavior.

Source and evidence access

This article summarizes the September 22, 2026 Ledger report Artifact Z22yVj, titled “CLI benchmark ten-task final exploratory results and verified evidence index.” Its immutable slug is Z22yVj-cli-benchmark-ten-task-final-exploratory-results-and-verified-evidence-index, revision 1, with an 18,634-byte Markdown payload and SHA-256 c6125cb69fd7a17434c3d5e15d3e9dd3cc6b63e2c6337794075e286e260a67b7.

The report's evidence index identifies the prompt-delivery audit as UBV1Je, the ten-task comparison as iEf69r, cumulative cost coverage as 2uYKQe, and corrected evaluations as A5DK8k, hGBp0G, and 8YgHFJ. These are Ledger identities, not public download links. Readers with access to the study's Ledger can inspect the report metadata with yy ledger artifact get Z22yVj -f json; this post does not claim public access to raw sessions, repaired byte-export support, or independently reproducible access to the private evidence archive.