A bounded longitudinal coding-agent benchmark dataset

Researchers who want to study how coding-agent systems have performed over time usually face a rebuild: pick a public leaderboard, scrape it once, normalize the rows, write down what was dropped, and start the clock on their own copy going stale. This page removes that step for one well-defined population. It publishes a bounded longitudinal dataset — 180 dated entries from the public SWE-bench Verified leaderboard, each one an evaluated agent system attempting the same fixed set of 500 real GitHub issue tasks — spanning 2023-10-10 to 2026-02-26. The extract is downloadable as a 146,765-byte JSON document, versioned as dataset version 1, and pinned by a content hash over its entries so any citation can name exactly the bytes it leaned on. What the series can support: the movement of publicly submitted best-effort results, the mix of attempt policies and scaffolds behind them, and the practice of reporting cost and verification. What it cannot support: a census of capability, a model ranking, or a prediction about any particular repository — the limitations section owns that boundary in detail.

Provenance first, because a dataset is only as good as its honesty. The leaderboard page was fetched from its public surface on 2026-08-29; that page's server stamp read last-modified 2026-08-10, and the bytes were identical to a fetch made one day earlier, so the snapshot is stable at least day-to-day. Every quotation below was matched word-for-word against the fetched page on the same date. Every count, date, and figure stated in this guide recomputes from the published JSON — none of them comes from memory of the leaderboard. The entries are third-party, self-reported results; nothing in the dataset measures any YYLO product, and the extract exists for citation and analysis with attribution to the surface it came from.

One board, one task set, thirty months

The population is deliberately narrow, because longitudinal claims need a fixed yardstick. SWE-bench Verified describes itself as "a human-filtered subset of 500 instances from SWE-bench, created in collaboration with OpenAI", and the human filtering is concrete: "Human annotators reviewed each instance to ensure the problem descriptions are clear, the test patches are correct, and the tasks are solvable given the available information." The unit on its leaderboard is not a model but a system: "The Verified leaderboard features results from a wide variety of AI coding systems, from simple LM agent loops to RAG systems to multi-rollout and review type systems." One task set, frozen; many systems, submitted over time — which is exactly the shape a longitudinal extract needs.

Each of the 180 entries is one evaluated submission: a display name, an entry date, the agent (scaffold) and organization behind it, the model or models it wore, an attempt policy tag, the percentage of the 500 instances it resolved, sometimes costs, and the board's verification marker. The source_folder field carries the entry's identifier on the source surface, so every row maps back to the materials behind it. Four entries date to 2023-10-10 — the retrieval-augmented baselines from the original benchmark paper, resolving between 0.4 and 4.4 percent. From there the stream grows: 4 entries in 2023, 51 in 2024, 112 in 2025, and 13 in early 2026, submitted under 77 distinct agent names. Composition matters as much as count: 116 of the 180 entries run open-source agent systems, and 25 of them pipeline more than one model.

What the series shows

Read chronologically, the resolved-percentage frontier climbs in steps that are themselves a record of the field's arrival. The 2023 baselines top out at 4.4. An entry first crosses half the task set on 2024-10-28 at 50.0, reaches 62.2 on 2024-12-21, touches 70.0 on 2025-04-30, and tops out at 79.2 on 2025-12-05. The 2026 cluster is different in kind: all 13 entries dated 2026 are frozen-scaffold runs, and their best reads 76.8 on 2026-02-17. That frozen-scaffold view is the board's own instrument for model-side comparison — "While the full leaderboard compares arbitrary systems, we are also interested in evaluating language models directly", with every model placed in the same wrapper: "we evaluate all LMs using mini-SWE-agent in a minimal bash environment. No tools, no special scaffold structure; just a simple ReAct agent loop." The dataset marks those runs: the 47 entries carrying an agent_version value are precisely the frozen-scaffold population, interleaved with the arbitrary-system entries on one board.

The reading discipline is the point. This is a stream of submissions, not a designed sample: the frontier records what someone chose to submit (and in most cases did not get checked), so the series measures public best-effort attempts, not the state of the art and not the average system. The composition fields make the honesty cheap: 84 of the 180 entries carry an attempt policy of 2 or 2+, 45 publish a cost figure, and only 60 carry a verified marker of true. Any trend line drawn over these rows inherits every one of those caveats — which is why the dataset ships them per entry instead of averaging them away.

Methodology: selection, not measurement

The extract is a selection from a public page, and every step is deterministic. The leaderboard document was fetched from https://www.swebench.com/ on 2026-08-29 (server last-modified 2026-08-10); the SHA-256 of the fetched document is recorded in the JSON alongside the fetch date, so the input is provenance-bound exactly the way this site's keyword evidence is. The board data arrives as an embedded JSON document inside the fetched page; the extractor parses it, selects the Verified board, and normalizes each entry's scalar fields. The other boards are excluded on purpose: the lite, multilingual, and multimodal editions are different task populations, and mixing them would break the fixed-yardstick property; the frozen-scaffold view is not a separate population at all, since its 47 entries already sit inside this board carrying the agent_version marker.

Normalization is minimal and documented in the file's field_notes. Numeric values are republished exactly as they appear — no rounding, no imputation — and absent values stay null rather than becoming zero. Exactly two normalizations transform anything: the model release date is rewritten from the source's compact YYYYMMDD form to ISO, and the verification marker, when the source states it in note form, becomes verified: false with the note preserved verbatim in verified_note. Order is preserved as published (resolved percentage descending, ties in source order). Dropped on purpose: logos, trajectory and log links, and the bulky per-instance detail objects that only some entries carry — the extract stays a 146,765-byte document rather than a mirror. The generator is committed in the same repository as the dataset, so the pipeline can be re-run against a fresh fetch:

curl -fsSL https://www.swebench.com/ -o swebench.html
npm --prefix frontend run seo:dataset -- --input swebench.html --fetch-date 2026-08-29 --last-modified 2026-08-10
npm --prefix frontend run seo:dataset:check

The check command re-validates the committed extract fail-closed: it recomputes the entries hash, verifies the population bounds and per-field shapes, rejects duplicate source folders, and confirms the published ordering still holds. A refreshed extract that disagrees with its own header fails instead of shipping.

Limitations

• Self-reported submissions. Submitters run their own systems and report their own numbers. There is no sampling design, so the absence of a system means the absence of a submission, never the absence of capability.
• Entry dates are listing dates. A run precedes its listing by an unknown margin; ordering the series by entry_date orders publication, not measurement.
• Attempt policies differ. 84 of 180 entries carry a 2-or-more attempt policy against 96 single-attempt entries. A resolved percentage earned over repeated tries and one earned on a single pass are different quantities; the tag travels with every row so no analysis is forced to blend them.
• Scaffolds differ. The full board compares arbitrary systems by design; model-only comparison belongs to the frozen-scaffold view, and even that view warns across its own release generations: "Results of release 1.x and 2.x are not necessarily comparable to each other, as 2.x uses tool calling to invoke actions, whereas 1.x parses action from the output strings."
• Contamination is unmeasured. The task set was frozen in 2024, and models released later may have trained on public material overlapping it. Later entries are not timed against equal exposure, so calendar progress and information advantage are confounded by construction.
• Verification is mixed. 60 entries carry a checked-true marker, 103 read false — six of them in the source's own words as "false (See README.md for info on how to get your results verified)" — and 17 carry no marker at all. The field is preserved; nothing in the extract upgrades an unchecked row.
• Cost coverage is partial. 45 of 180 entries publish cost, spanning $0.00 to $765.57, with no adjustment for provider price changes across the series. Missing cost stays null; it is never zero.
• Identity is as declared. 35 entries carry no model tag — 24 of them display an undisclosed model — and ensemble membership lives in the models array, which sometimes names models the display name omits. Identities are submitter-declared, not audited.
• Snapshot semantics. The leaderboard is live; entries can be added, edited, or retracted after this fetch. The dataset version, fetch date, source hash, and entries hash pin exactly what was observed on 2026-08-29, nothing more.
• Not YYLO measurements. The entries belong to their submitters and the benchmark team; this extract wraps their public surface for reuse with attribution, and no figure here describes any YYLO product.

Versioning and citation

The extract is content-addressed the way evidence should be: entries_sha256 is sha256:cbd2393a235c0276211fb4d7035c943ef7eb7e4be4902be99a5733f6380d02b6, computed over the entries array in a documented canonical form (recursively key-sorted JSON, no whitespace). Dataset version 1 is never silently rewritten: a refresh re-runs the committed generator against a new fetch, publishes a new dataset version with a new fetch date, source hash, and entries hash, and leaves the old version's claims intact. Cite the dataset by this page's stable URL (printed in the panel below), the dataset version, and the entries hash; cite the file by its download URL. The source surface asks for the same courtesy in kind — "If you use SWE-bench Verified in your research, please cite our paper" — and since this extract is a wrapper around that surface, a citation that names both is the correct one. Versioning and citation policy for recurring research assets across refreshes is treated site-wide in the recurring-research methodology guide.

Where this dataset sits

This page owns one artifact and its honesty: a bounded, versioned, downloadable longitudinal extract with stated limitations. Around it, each neighboring question has its own owner. Which family of instrument a leaderboard belongs to — arena, frozen public suite, index, usage ranking — and what each family's numbers are entitled to decide is the instrument reference. Why any public standing transfers weakly to outcomes inside one specific repository is the transfer analysis. The statistical arithmetic of repeated attempts — intervals at small counts, what a rate proves — is the variance guide, and one boundary holds between these pages: that guide's figures are model-based arithmetic computed from committed formulas, while this dataset is empirical third-party listings, and nothing here confirms or calibrates those figures. Repeated-attempt data as a dataset in its own right awaits a dedicated treatment.

The same longitudinal question, asked about evidence you control, has a product-side answer: YYLO Benchmark's report layer unions compatible retained attempts across experiments while preserving provenance — the mechanics are documented in the methodology reference, and the operator path for producing such evidence on a real repository is the walkthrough. The product behind that machinery is YYLO Benchmark. If the extract above saves you a scrape, cite the stable URL in the panel below, name the version and hash, and let the limitations section travel with the numbers — that is the entire contract this dataset offers.