Coding-agent benchmarks vs model leaderboards

Every week crowns a new top of some leaderboard, and every week somebody reads that crown as an answer to a question the leaderboard never asked. The searches that arrive here — "llm leaderboards", "ai leaderboard", "lmarena leaderboard", "open llm leaderboard", "grok 4 benchmarks" — are model-side questions: which model ranks where, on whose numbers. The decision waiting behind those searches is almost never a model-only decision. It is whether a model, inside a harness, with tools and instructions and a budget, resolves the kind of work one specific repository contains, repeatedly, at a price the team accepts. Those are two different measurements. This page is the reference that keeps them apart: what each instrument actually measures, why the numbers diverge even when both are honest, and which decision each kind of evidence is entitled to support.

Every quote below was pulled from its linked surface on 2026-08-28 and verified verbatim — the arenas, suites, indexes, and usage rankings all described in their own words — and every claim about repository-level evaluation against the committed guidance of YYLO Benchmark. The date belongs to the evidence: every surface cited here changed before and will change again, which is itself the first reason a leaderboard needs to know what it is measuring.

Four instruments wearing one name

Preference arenas. LMArena, whose homepage titles itself "Arena AI: The Official AI Ranking & LLM Leaderboard", ranks models by community votes cast in pairwise battles — its own navigation is built around voting and battle modes — and its overview invites you to "See how leading AI models stack up across text, image, vision, and more." Every entry carries a score with a confidence interval rather than a bare number, and the battles behind those scores are conversations, not repositories. What an arena measures is which answers humans prefer when two models respond to the same prompt — a real signal, one step removed from any engineering task.

Fixed public suites. SWE-bench is, in its own caption, "The original benchmark: real GitHub issues from 12 Python repositories." Its human-filtered edition is "A human-validated subset of 500 SWE-bench instances for reliable evaluation of coding agents and language models." Aider's leaderboard takes a different task family — "Aider’s polyglot benchmark tests LLMs on 225 challenging Exercism coding exercises across C++, Go, Java, JavaScript, Python, and Rust." — and the page says plainly that Aider "uses benchmarks to evaluate an LLM’s ability to follow instructions and edit code successfully without human intervention". LiveCodeBench "collects problems from periodic contests on LeetCode, AtCoder, and Codeforces platforms", and because contest problems are dated, "LiveCodeBench annotates problems with release dates" so a model can be scored on problems published after its training cutoff. Terminal-Bench widened the aperture again: "terminal-bench is a collection of harbor-native benchmarks to help agent makers quantify their agents' terminal mastery". A suite measures performance on a task set its authors froze in public.

Indexes and aggregators. Artificial Analysis "performs intelligence, quality, performance and price benchmarking on AI models, inference API endpoints and systems" — many underlying measurements weighted into one index, with price and speed reported beside it. Vellum's table states its sourcing outright: "The data comes from model providers as well as independently run evaluations by Vellum or the open-source community." and "We feature results from non-saturated benchmarks, excluding outdated benchmarks (e.g. MMLU)." An index measures a weighted blend of other people's measurements.

Usage rankings. OpenRouter's rankings bill themselves as "Live LLM rankings by real-world usage. Models are ranked by tokens processed through the OpenRouter API by millions of developers." OpenRouter then draws the boundary itself, in a section titled after what the numbers do not establish: "These rankings measure adoption, not quality." and "They describe traffic routed through OpenRouter, not the whole market and not usage on a model provider's own API."

Four instruments, four measurements — preferred answers, frozen public tasks, blended indexes, routed tokens. What they share is the shape: the unit of analysis is a model, the task set or vote pool is shared and public, and the output is one comparable score per model. That shape is exactly right for ranking models. It is exactly wrong for the repository question, and the next section is where the seam shows.

The system is not the model

The clearest evidence that model and system are different units comes from inside a public benchmark itself. SWE-bench's full Verified leaderboard "features results from a wide variety of AI coding systems, from simple LM agent loops to RAG systems to multi-rollout and review type systems." — a comparison of scaffolds and models together. But when the same team wants to compare models fairly, it freezes the scaffold: "we evaluate all LMs using mini-SWE-agent in a minimal bash environment. No tools, no special scaffold structure; just a simple ReAct agent loop." One benchmark, two leaderboards, because a system and a model are different things to measure. The arena world feels the same pressure: in the WebDev arena fetched for this page, one top-ten entry carries the parenthetical "(codex-harness)" — when the score depends on the wrapper, the wrapper gets named.

A repository-level coding-agent benchmark takes the systems view all the way into your work. The unit is a model inside a harness attempting a real engineering case from your repository; the measure is whether the attempt resolves, repeated across attempts; and the package's own overview commits to the question up front — "which model or agent system is the least expensive option that resolves a category of real engineering work reliably enough?" Cases come from commits you can rebuild, each dispatch leaves from one identical exported snapshot, the grader is bound to the exact bytes of an executable, and the run leaves receipts rather than a leaderboard row. The full operator's path through that machinery — case authoring, the frozen attempt matrix, spend ceilings, isolated execution, grading — is the real-repository guide; this page stays on the distinction.

Five reasons the numbers diverge

• The task distribution is not your repository. A suite samples public GitHub issues, contest problems, or exercises; your repository has its own languages, conventions, test quality, and change sizes. Public suites also fight training contamination — that is precisely why LiveCodeBench stamps release dates on problems. Your unpublished work cannot have leaked into a training set, but it is also not the suite's task mix, in either direction.
• The harness is a variable you stopped controlling. On a leaderboard, the scaffold is fixed or vendor-supplied so the model can be isolated. In your work, the harness is a choice you own — tools, context budget, iteration bounds, review steps — and changing it changes outcomes on its own, which is exactly why the benchmark teams freeze the scaffold before comparing models at all.
• One number hides the dimensions that decide. Leaderboards collapse to a rank. The repository question refuses the collapse: "Keep cost, elapsed runtime, invalid-run rate, and repeat consistency as separate dimensions rather than collapsing them into a universal score." A model can be fast and unreliable, cheap per token and expensive per success; the single score has nowhere to put that.
• One attempt is a coin flip; reliability is a rate. Aider's own detail rows carry "Pass rate 1" and "Pass rate 2" side by side, and one entry observed on 2026-08-28 scored 52.0 on the first attempt and 88.0 with a second — a thirty-six point gap from retry alone. Repository guidance is built on the denominator instead: "Compare repeated valid attempts to estimate reliability." and "Include sample size and uncertainty; small repeated samples support cautious recommendations, not claims of universal superiority."
• Someone must own the failures that are not the model's. On a public board, the environment is the operator's problem and mostly invisible. In your repository, the environment is your problem, and misattribution corrupts the record in both directions: "Infrastructure-invalid attempts remain visible in invalid-run statistics but do not become model failures." A flaky runner does not make a model worse, and a heroic runner does not make it better.

None of these reasons is a complaint about leaderboards. Each instrument is measuring what it said it would measure. The divergence appears only when a model-side number is asked to answer a system-side question.

Which decision each kind of evidence supports

Shortlisting frontier models for general quality is an arena question; shortlisting them for capability on frozen public tasks is a suite question; blending capability with price and speed is an index question; sensing what the field actually routes today is an adoption question, best read with OpenRouter's own caveat attached. None of those four decisions is the one your repository is waiting on.

The repository decision — what to run on your code, at what cost, with what confidence — belongs to evidence produced inside that repository: resolution rates with sample sizes, cost per successful resolution, runtime, and failure classes, each retained as checkable evidence. The cost line deserves its own sentence, because cheap is the easiest number to abuse: "Report cost per successful resolution alongside raw cost and reliability so cheap unresolved work is not rewarded." And the output stays a recommendation, not a config change: "Recommendations are advisory: they do not mutate production routing or deploy candidate patches."

The two stages compose honestly. A leaderboard is a fine instrument for drawing the shortlist of candidate models; the repository benchmark is the instrument for confirming which of them actually does the work. Teams get hurt when they swap the order — adopting a leaderboard verdict as a deployment decision, or demanding repository-grade certainty from a public board that never saw their code.

The demand behind the vocabulary

The competitor exports behind this site's keyword coverage, dated 2026-08-26, carry forty-three keyword phrases across forty-nine rows for this intent, an estimated 28,850 combined. The generic-noun half leads: "llm leaderboards" (8,100), "ai leaderboard" and "ai rankings" (3,600 each), "ai model rankings" (1,900), "llm model ranking" (590), "model ranking" (210). The arena family sums to 2,670 across its three spellings — lmarena (1,600), lmsys (590), chatbot arena (480). The open/open-source/uncensored cluster runs 590, 390, and 390, with "hugging face ai leaderboard" (30) and "open source model ranking" (40) beside it. Named instruments appear as themselves: aider (390), openrouter (260), artificial analysis (300 across two phrasings), vellum (210), plus "embedding model leaderboard" (210) and "llm coding leaderboard" (210). The merged half is model-specific lookup — "grok 4 benchmarks" (1,000), "ai benchmark ranking" (880), "llm coding benchmarks" (390), and glm, fable, gpt, deepseek, kimi, and gpt-oss phrasings between 40 and 320 each — searchers arriving for one model's numbers. This page is written for exactly that arrival: the numbers they want are model-level measurements on public instruments, and the sections above say which decisions those numbers may and may not carry. The ranking footprint adds its own wrinkle: across these forty-nine rows, openrouter.ai holds twenty-seven positions, coderabbit.ai nine, z.ai five, opencode.ai four, every.to two, and cursor.com two — and none of the arena, suite, or index surfaces quoted above appears in this footprint at all. A fresh pass over the committed dispositions found no phrase containing "yylo". Each figure in this paragraph is a third-party demand estimate drawn from this site's 2026-08-26 competitor keyword exports — it counts lookups, not outcomes, and forecasts nothing about traffic or quality — and the paid competition metric never informed a sentence here.

Where this page sits

This page owns one distinction — what a model leaderboard measures, what a repository-level coding-agent benchmark measures, why they diverge, which decision each supports — and hands every neighboring question to its owner. The operator's path through an actual repository run is the real-repository guide. Which harness wraps which model is a comparison question with its own dated harness field matrix. Which stack layer owns evaluation is mapped layer by layer in the stack survey. The narrower statistical question — whether a public standing predicts outcomes inside one particular repository — is a subject of its own.

If this reference saved you an argument, cite it by the stable URL in the next-step panel below. Every quote carries its surface and its date, and the distinction it defends is the one the instruments themselves keep drawing in their own words.