Why public leaderboards may not predict repository outcomes

The scene repeats weekly. A team reads a coding leaderboard, adopts the model sitting at or near its top, wires it into their agents — and some weeks later the wins in their repository are smaller than the board implied and the failures arrive in places the board never mentioned. The board did not lie. It answered, carefully and on its own terms, a question nobody on the team asked out loud: how does this model rank on that instrument's public task set, under that instrument's fixed conditions, at that instrument's sample sizes? This page is about the question the team actually had — will it resolve the work in this repository, repeatedly, at a cost we accept? — and about why a public standing carries over to that answer only weakly, for three structural reasons no amount of leaderboard refinement removes.

One neighbor owns half the vocabulary. Which instrument a number came from — preference arena, frozen public suite, blended index, or usage ranking — and which decisions each kind of number may support is the job of the instrument reference. This page owns the transfer question itself. Its contribution is an analysis in three mechanisms — the task distribution, the harness, and sampling variance — plus a decision rule stating when a standing may still guide you and when it must be demoted to a prior. The analysis and the rule are this site's original work, versioned by the date below; each quotation was pulled fresh from the surface it names on 2026-08-28 and matched character-for-character against that page's text. The boards recompute continuously — the usage ranking stamps the very page fetched for this guide "Usage data through Aug 27, 2026" — so specific standings will have moved by the time you read this, while the three mechanisms stay put, because they belong to the shape of public measurement, not to any particular board.

The task mix is not your repository

Each suite scores a public sample of work, and the sample has a shape. The SWE-bench family page describes its own members with counts: the multilingual edition carries "Tasks from 42 repositories across 9 programming languages.", the lite edition is "A subset curated for less costly evaluation.", and the original benchmark pulls its issues from twelve Python repositories. Two consequences follow. First, the family's own analysis views slice results "Resolved by repository" and "Resolved by language" — the maintainers themselves treat resolution as something that varies by repository, not as one number. Second, every member is a mixture with its own languages, test quality, and change sizes; your repository is a mixture none of them sampled. A standing is an estimate over their mixture, and your work is a draw from a different one.

The sharpest published evidence that ordering flips with the mixture comes from LiveCodeBench, reporting its results across scenarios: "Claude-3-Opus overtakes GPT-4-turbo in the test output prediction scenario, but not in the code generation scenario." When relative order swaps between two scenarios of a single benchmark, nothing protects it from swapping between that benchmark and your checkout service. A repository is a scenario — its languages, its conventions, its test depth — run by nobody before you.

Contamination widens the gap in the other direction. Public suites fight leakage because public tasks can be trained on; LiveCodeBench introduces itself as "a holistic and contamination-free evaluation benchmark of LLMs for code that continuously collects new problems over time", and its own finding shows why that fight is necessary: "We find that DeepSeek models exhibit a stark drop in performance on LeetCode problems released since September 2023, its release date, indicating that the earlier problems might be contaminated." Your repository is unseen in both directions — no model memorized its answers, and no suite measured its tasks. The protections that make a board honest about public work buy you nothing on private work, and the coverage was never claimed.

The harness is not your harness

To isolate the model, a board must control the scaffold the model runs in. SWE-bench says so in the tooltip of its own environment toggle: "Show only runs in the mini-SWE-agent environment, so scores compare models rather than harnesses". Read that label in reverse and it confesses what the filter holds back — with the toggle off, the board compares harnesses. The team went further and made that view the standard one; the family page's card for it reads "The default Verified view: every model in the same mini-SWE-agent environment." And the scaffold variable is not a rounding error of model choice; the same site's news archive records that "mini-SWE-agent scores 65% on SWE-bench Verified in 100 lines of Python." A hundred-line wrapper produces that resolved rate on the same five-hundred-instance set the Verified boards score — the size of the wrapper does not bound the size of its effect, which is why the fair view freezes the harness before comparing models at all.

Aider's leaderboard pins its harness state on every row. Each record carries its own commit hash, edit format, and run date, and the two gpt-5 rows read for this page share one benchmark commit while differing in reasoning effort and two days of calendar: one resolves 88.0 percent of exercises for $29.08, the other 86.7 percent for $17.69. Settings within a single harness — not model choice — moved both the score and the bill. Now run the implication into your repository: there, the harness is yours — your tools, your context budget, your iteration bounds, your review gates — and the board's number was produced under scaffolds you are not running. The outcome you care about comes from a system the board never measured, composed of a model it did measure wearing equipment it did not.

One number, one attempt, one draw

The third mechanism is the plainest: single numbers from small samples are draws, and boards publish the evidence for this in their own rows. Aider's detail records carry both attempts side by side. The o3-pro row on the leaderboard read for this page shows "Pass rate 1 : 43.6" and "Pass rate 2 : 84.9" — a 41.3-point gap from retry alone, nearly doubling the resolved fraction, on a run whose cost line reads $146.32. The headline "Percent correct" column, for each of the three rows examined here, prints the second-attempt figure. If the number you carried away from the board was one attempt and the number the board printed was two, the gap between them was already hiding inside the prediction you imported.

Arenas publish their uncertainty the same way. The agent table on the arena board fetched for this guide tops out with "12.99%±1.71%" just above "12.73%±1.96%" — bands that overlap. Another table on the same page, the overall text standings, opens with five rows at "1507±5", "1505±4", "1502±4", "1498±10", and "1497±3" — a chain in which every adjacent pair of bands overlaps. Adjacent ranks on these boards are ties wearing ordinal numbers, and the boards themselves say so through the bands they attach.

Repository outcomes compound the problem because they are rates, not points. One dispatch is one draw; a resolved rate over repeated attempts is the estimand, and your real work adds variance the boards never modeled — flaky runners, quota stalls, reviewer latency, merge conflicts. A two-point win inside a four-point band is noise with a medal pinned on it. The benchmark guidance this site publishes elsewhere exists for exactly this reason: reliability is a denominator question, and denominators do not fit in a rank.

A three-question rule for trusting a standing

The mechanisms above do not say leaderboards are useless; they say a standing is a conditional kind of evidence. Before letting one influence a decision, answer three questions in order.

First: what does the number measure? Name the instrument and its unit of analysis — community preference votes, frozen public tasks, a weighted blend, or routed tokens — and name whose conditions produced it. If you cannot say, the number cannot transfer. And if the honest answer is adoption, the ranking service itself draws the boundary, in a section it titles after what its numbers do not establish: "Token volume is also not a count of requests, users, or spend, and models differ in verbosity and tokenization, so a higher token total does not by itself mean a model is preferred for a given task."

Second: is the measured work shaped like yours? Compare the mixture to your repository — languages, test quality, typical change size, conventions — and say the similarity out loud. The SWE-bench family slicing its own results by repository and language is the maintainers' acknowledgment that resolution varies with the work's shape; if your repository's shape is not represented in the task set, the standing is a prior about models in general, not a measurement of your work.

Third: does the margin clear the noise? Find the bands, the attempt counts, and the data date. Overlapping bands make ties; single-attempt scores are samples of one; and a board stamped with an old data window is quoting a past season of a moving estimate. A candidate must clear the combined noise before the rank difference means anything.

Three passes and the standing may legitimately narrow a shortlist of candidate models — that is the decision leaderboards are built for. Any failure and the standing is a prior, not a prediction, and the decision needs evidence produced closer to the work. Notice what the rule never asks: whether the board is any good. The boards are good. The question is transfer, and transfer is decided by the work's shape, the scaffold, and the arithmetic of the margin — all three of which live on your side of the fence.

What settles it inside one repository

The minimal honest measurement is small in machinery and heavy in discipline. Pick one genuine item of engineering work from the repository you are deciding about. Attempt it repeatedly from identical exported baselines, so whatever varies between attempts is attributable to the candidate rather than to the baseline. Grade with a command whose verdict does not depend on the candidate's own testimony. Report the resolved rate with its attempt count attached; attach cost to successes rather than to tokens; and keep a failure ledger that separates environment faults from candidate faults, so a flaky runner cannot libel a model and a heroic one cannot launder it. Then keep the records, so the next board refresh cannot retroactively rewrite what happened in your code. The operator path that builds exactly this evidence — case selection, frozen attempt matrices, isolated execution, hash-pinned grading, retained receipts — is the real-repository guide. Two questions that go deeper than this page — how repeated attempts expose variance, and how to choose an agent by repository and change type — each now has a page of its own: the variance treatment and the selection method.

Where this page sits

This guide holds one question and its answer: why a public standing transfers weakly to one repository's outcomes, and when it may still be trusted. What each instrument measures and which decision each supports belongs to the instrument reference; producing the repository-side evidence belongs to the real-repository guide. If the three mechanisms or the three-question rule settle an argument your team was about to have twice, reuse them with attribution and cite this page at the address printed in the next-step panel that follows the body. The mechanisms are structural and will outlast any given board; the standings quoted here are dated 2026-08-28 and will have moved since — which is, one last time, the point: a standing is an estimate about someone else's work, and your repository deserves its own.