Choose a coding agent by repository and change type

"Which coding agent should we use?" is the question teams ask first and the question evidence answers last. It is also two questions wearing one sentence. There is the brand question — which agent is best, in the abstract, this quarter — and there is the work question: which agent will resolve this kind of change, inside this repository, at the risk this repository can absorb, at a price this repository accepts. The brand question is settled by fashion and re-settled by the next release. The work question is settled by measurement, and it stays settled until the work itself changes shape. This page is a method for the work question: classify the repository and the change, benchmark one representative case for that classification, and read the result as a portfolio — a default candidate, an escalation candidate, and a written bar both must clear.

Three neighbors hold the surrounding ground, and each is linked once below where its job begins. Producing the evidence — choosing a real engineering case, freezing its plan, running isolated attempts, grading them against pinned bytes — belongs to the operator walkthrough. Why a public standing forecasts someone else's repository weakly, and when it may still narrow a shortlist, is the subject of the transfer analysis. Harness against harness, product against product — Cursor, Kiro, OpenCode, Pi, Claude Code, Codex, YYLO — stays at the comparison matrix. What remains is the layer between them, the one no other page owns: given the work, how the pick is actually made. The five quotations in this guide were read from the benchmark package's committed guidance pages on 2026-08-29 and matched byte-for-byte after whitespace normalization; the classification scheme and the portfolio reading are original to this site and carry the evidence date above as their version.

Classify the work before naming any candidate

Most agent debates are unwinnable because they are fought over the wrong nouns. "Is A better than B?" has no referent until the work is fixed; the same pair of agents can reverse order across two repositories, and neither ordering is wrong. So the first move of the method names no candidate at all. It fixes three properties of the work, because those properties — not the agents — decide what kind of evidence the pick will need.

Repository size, for selection purposes, is reachability: how many places a plausible change must touch before it is done, and how much convention and history an agent has to carry to move through them without breaking anything on the way. A small repository forgives an agent that reads narrowly. A large one punishes that same agent with plausible edits in the wrong dialect of the codebase — each edit individually defensible, the sum of them a rejection.

Repository risk is blast radius: what breaks if the change is subtly wrong, who notices, and how recoverable the damage is. A prototype and a payment path can present identical diffs; one's failure is an afternoon, the other's is an incident. Risk decides how much reliability the cell must buy before economy is allowed to matter at all.

Change surface is the shape of the edit itself. Narrow and mechanical: one behavior, one deterministic check that says done. Wide and mechanical: one transformation, many files, the same judgment repeated at scale. Deep and exploratory: migrations, redesigns, new integrations — work where the right shape is discovered while the work proceeds, and where a fluent first draft is precisely the failure mode.

Agents do not wear these properties equally, and that is the whole trouble with brand loyalty. Strength on one axis is routinely weakness on another: the agent that is unbeatable inside a bounded refactor can be uneconomical on a wide sweep, and the agent that cheerfully sweeps can be dangerous in the deep end. No ordering of candidates survives contact with all six cells below — which is the reason cells exist.

A matrix that names a posture, not a winner

size    risk    surface             selection posture               evidence to weight
small   low     narrow mechanical   economy first                    cost coverage, resolved rate
small   low     wide mechanical     context capacity, then economy   resolved rate, runtime
small   high    any                 reliability before economy       typed failures, resolved rate
large   low     narrow mechanical   navigation economy               runtime, cost per resolved outcome
large   low     wide mechanical     accuracy at scale                repeat consistency, invalid-run share
large   high    deep exploratory    strongest evidence, human gate   consistency across repeats, failure classes

Read the matrix as a posture assignment, never a product assignment. Each row names the dimension that decides its cell and therefore what a benchmark for that cell must report cleanly: resolution rates plus the attempt counts behind them, runtime distribution, failures separated into typed classes, and what the runs cost. The right-hand column is the discipline — a cell that cannot name its deciding dimension is not yet a decision; it is a preference.

One row deserves a sentence because intuition gets it backwards. The small-and-high-risk row outranks economy entirely: when the blast radius is real, the cheapest candidate is the expensive one, because an attempt that resolves wrongly on a risky path costs more than the tokens ever could. And notice which combination is missing — deep exploratory work in a low-risk repository is usually a sign the risk was misread; find what the change actually touches before trusting that label.

One case per cell, and the cell's case only

Evidence for a cell comes from that cell, or it is not evidence for the cell. The benchmark package's authoring contract opens with the selection discipline in one breath: "A benchmark case is a real engineering item selected through explicit owner opt-in. Do not infer eligibility from labels, age, or convenience." Read the last three nouns as a portrait of default agent choice. Labels: it is the one the team already pays for. Age: it has been the default since spring, which is tenure, not a measurement. Convenience: it was installed first. Every brand-loyal pick is one of the three wearing a logo.

The case must then speak with one checkable voice: "State one primary resolved outcome with deterministic evidence where possible." A representative case that resolves to "looks reasonable" has produced a preference; the deterministic half is what lets a stranger — or next quarter's you — re-run the judgment without re-litigating it. And where no such case can be built, the authoring checklist is blunt about the consequence: "Reject cases whose correctness cannot be judged or whose baseline cannot be rebuilt." Selection inherits that refusal honestly: a cell with no judgeable case has no evidence, so its pick is a prior — provisional, cheap to abandon, labeled as such — until the work grows a check or the check is written first.

One more authoring rule keeps cells representative. The case must be drawn from the work the cell names: a benchmark of typographical fixes selects an agent for typographical fixes, whatever the benchmark's prestige, because prestige earned inside one cell is not evidence inside another.

Read the result as a portfolio, not a podium

The reading rules are as mechanical as the authoring ones. Write the bar first: the reliability a cell demands — how often a change of that kind must land correctly to be worth dispatching at all — goes on paper before any candidate runs, because a threshold negotiated after the standings exist is a preference in costume. With the bar fixed, the output is two names, not one. A default: the least expensive candidate whose measured rate on that cell's case clears the written bar. An escalation: one stronger candidate held for the dispatches where a miss costs most — the risky cell, the release-week cell — named in advance, so it is a policy rather than a rescue.

The arithmetic that keeps those two names honest lives in attempt counts and intervals; this program's treatments of repeated attempts and of several agents sharing one task cover it, and selection only inherits the consequences. Narrow evidence makes wide claims impossible — three attempts per candidate sustain a cautious default, never a coronation — and two candidates whose results overlap are a split verdict, not a tie to break by taste: the cheaper name takes the default slot, the stronger one stays the escalation, and the cells decide which name answers which dispatch.

Money gets one rule and it is absolute: "Report missing or partial economics explicitly." A comparison of two candidates' costs when only one candidate's costs are complete is not a comparison; it is a discount awarded to whichever bill went missing. Unpriced work is the most expensive work — cost belongs inside the verdict, marked incomplete where it is incomplete, so the default slot never goes to the candidate whose evidence simply stopped.

Re-select when the repository moves

A selection is evidence wearing a date, not a property of the team. Repositories drift in every direction: conventions harden, test depth grows, a migration redefines what a wide mechanical change even touches, and the cell map itself moves underneath the portfolio. The package handles the within-case version of that drift with a flat rule: "Increment the case version when intended meaning changes." Portfolio selection runs the same rule one level up. When a cell's meaning changes, the cell's evidence retired with the old cell: re-measure the new cell against the same written bar, keep the old results as history, and never carry a recommendation across a change in the work it measured. A default that survives its repository's migration untouched is not durable; it is unexamined.

Where this method sits

This page's own job is the mapping from repository size, risk, and change surface to a defensible pick: the posture per cell, the evidence each posture must read, and the two-name portfolio with its written bar and explicit economics. Producing that evidence is the operator walkthrough's loop through a real repository; running several agents over one case fairly is the bake-off design; what repeated attempts do to small samples is the variance treatment; why borrowed standings mislead is the transfer analysis. None of it is restated here. Classify the work, benchmark one case per cell, write the bar before the runs, and let the least expensive candidate that clears it carry the default load — the install panel below brings the tool that produces this evidence.