Tasks that have already beaten a model
Every task below was run under gate 5 before it was listed, and carries the pass rate that round measured. Open one to see which model failed it and by how much.
58%
7 of 12 trials
Rebuild a settlement ledger against withholding mints
Software engineering · medium tier · python
A venue settles trades against quote pools, and the ledger has to be rebuilt from state so it can be reconciled against the chain. Some quote assets are tokens whose mints withhold part of every transfer and do not publish their terms, so the terms have to be inferred from the transfers that were made and what was kept out of them.
- GPT-6 AstraTerminus 24 of 4
- Claude Fable 5.1Terminus 23 of 4
- GPT-5.3 CodexCodex CLI0 of 4
GPT-6 Astra passed 4 of 4. GPT-5.3 Codex passed 0 of 4. The headline above reports both of them as the same task, which is why every row in the catalogue opens onto its own breakdown.
Open the full breakdownThe set, and the evidence it was measured with
- The graded task
- Its environment and verifier images, pinned by digest, exactly as they were run.
- The trial record
- Every attempt behind the figures beside this, the ones that errored included.
- What stays with us
- The instruction, the tests and the reference solution. A measurement you can read the answer to has stopped being one.
Each card carries the rate its round measured and the cells behind it. Open one for the full table, the declared tier and what a licence includes.
- Software engineeringhard tier
Decide what a command string would actually run
A command-approval hook sits in front of a shell and has to answer, before anything runs, which external programs a command string would start and whether any of them is destructive. The hook it replaces reads the string as text and keeps being got around by wrappers put in front of the real command.
0%0 of 4 trials- GPT-6 AstraTerminus 20 of 4
bash-exec-resolve · python, bash
- Data processinghard tier
Compose SSH command batches from a maintenance plan
A fleet tool turns a maintenance plan into the command batches it sends to one server over SSH. Compose them against a written spec, from a plan and a runbook that are supplied separately, so that sending a batch twice is not the same as doing the work twice.
11%1 of 9 trials- Claude Fable 5.1Terminus 21 of 3
- GPT-6 AstraTerminus 20 of 3
- Gemini 4 ArgonTerminus 20 of 3
remote-batch-compose · python
- Data processinghard tier
Take a leaver out of the team tables
An account leaving the company has to come out of the team tables. The rules were written over four years, they are dated, and the handbook says which of its own sections carry force and which are one person's notes.
22%2 of 9 trials- Gemini 4 ArgonTerminus 21 of 3
- GPT-6 AstraTerminus 21 of 3
- Claude Fable 5.1Terminus 20 of 3
identity-not-position · python
- Finance and taxationmedium tier
Withhold tax against a schedule only a service holds
Compute the tax to be deducted at source on a company's vendor payments. The rate schedule is not in the repository at all: an MCP server is the only authority for rates, thresholds and the amendments that moved them, so every figure has to be fetched rather than assumed.
25%1 of 4 trials- GPT-5.3 CodexCodex CLI1 of 4
tds-rate-service · python
- Software engineeringhard tier
Merge a coding agent's queued messages into one turn
A desktop app runs coding agents behind a per-agent message queue. Messages that waited together are still read apart, so one thought costs four turns and pays for the whole session four times. Merge what waited together, in the order it was written, without the queue losing its liveness when a turn does not start.
33%2 of 6 trials- Claude Fable 5.1Terminus 22 of 3
- GPT-6 AstraTerminus 20 of 3
queue-turn-start · python
- Data processingmedium tier
Reconcile a month's input tax credit
A finance team closes a credit run by comparing what suppliers filed against what the company itself recorded, giving each invoice a status and an amount. The rules are the company's own and they are written down across several files, not all of which are current.
33%3 of 9 trials- GPT-6 AstraTerminus 23 of 3
- Claude Fable 5.1Terminus 20 of 3
- Gemini 4 ArgonTerminus 20 of 3
gst-itc-reconcile · python, bash
- Securityhard tier
Close a replay gap in a TOTP verify path
A login flow verifies TOTP codes statelessly, so the same six digits submitted twice inside one step both pass. Make a code work once across every account, while the store of accepted codes stays bounded on a box that authenticates far more accounts than it ever sees twice.
33%2 of 6 trials- GPT-6 AstraTerminus 22 of 3
- Claude Fable 5.1Terminus 20 of 3
pyotp-replay-guard-skew-window · python
- Data processinghard tier
Decide which test results are evidence about production
A nightly suite runs against a test backend that is not the production one, and the two disagree in ways somebody has documented. Decide which of the suite's results say something about production and which say something only about the test backend.
38%3 of 8 trials- Gemini 4 ArgonTerminus 21 of 3
- GPT-6 AstraTerminus 21 of 3
- Claude Fable 5.1Terminus 21 of 2
backend-disagreement · python
- Local cloud emulationhard tier
Find the money in a payout run that reported clean
Nine merchants' payouts left their available balances on Sunday night and nothing arrived. The run reported 240 payouts and no errors, the morning reconciliation came back clean, and the bank's statement says it sent everything it was given, which is true. Work out what happened from the incident folder.
38%3 of 8 trials- Claude Fable 5.1Terminus 23 of 4
- GPT-6 AstraTerminus 20 of 4
payout-dispatch · python
- Data processinghard tier
Report who holds a resource's data after a chain of hand-offs
A fleet of resources is installed, handed between teams and sometimes reinstalled, and each holds data one team is custodian of at a time. What a hand-off does and what a reinstall does to that custody were written down at different times.
44%4 of 9 trials- GPT-6 AstraTerminus 23 of 3
- Gemini 4 ArgonTerminus 21 of 3
- Claude Fable 5.1Terminus 20 of 3
normalise-then-collide · python
- Software engineeringmedium tier
Plan which MCP tools a turn can offer
A turn can hand a model only so many tools. Built-in servers take part of a fixed budget and the configured servers share what is left. Decide which servers fit, given that a server which does not fit should not cost the turn the ones that would have.
50%3 of 6 trials- Claude Fable 5.1Terminus 23 of 3
- GPT-5.3 CodexCodex CLI0 of 3
mcp-tool-budget · python
- Software engineeringmedium tier
Report every file a reference library holds
Two bibliography reports went out wrong and both went out silently: a file a reference manager had exported in the wrong encoding was dropped, and nothing in the report said a record was missing. Report what the library holds, including the files that cannot be read.
50%3 of 6 trials- Claude Fable 5.1Terminus 23 of 3
- GPT-6 AstraTerminus 20 of 3
bib-library-guard · python
- Software engineeringmedium tier
Reconcile an inbox against responses that arrive out of order
A desktop inbox shows a change the moment somebody makes it, while the request is still in flight, and responses come back in whatever order the network gives them. The list has twice jumped backwards: an older response restored a finished row, and a change that failed stayed on screen as though it had worked.
50%3 of 6 trials- GPT-6 AstraTerminus 23 of 3
- Claude Fable 5.1Terminus 20 of 3
inbox-revision-fence · python
- Software engineeringmedium tier
Recover a session's context figure from its transcript
A compaction reminder is silent in long sessions and premature in short ones, and both faults are in the probe that reads a session's context size. The transcript is appended to while the session runs, it is read from the end rather than whole, and the usage block it carries reports the same cache tokens in two shapes.
50%3 of 6 trials- Claude Fable 5.1Terminus 23 of 3
- GPT-6 AstraTerminus 20 of 3
transcript-context-probe · python
- Data processinghard tier
Decide which imported fields were altered on the way in
An export of stored records was imported back into the system and the import path changed some of them in transit. Work out which fields were altered and which were always like that, against a handbook written over several years and dated accordingly.
56%5 of 9 trials- Gemini 4 ArgonTerminus 23 of 3
- GPT-6 AstraTerminus 22 of 3
- Claude Fable 5.1Terminus 20 of 3
import-exactness · python
- Software engineeringmedium tier
Rebuild a settlement ledger against withholding mints
A venue settles trades against quote pools, and the ledger has to be rebuilt from state so it can be reconciled against the chain. Some quote assets are tokens whose mints withhold part of every transfer and do not publish their terms, so the terms have to be inferred from the transfers that were made and what was kept out of them.
58%7 of 12 trials- GPT-6 AstraTerminus 24 of 4
- Claude Fable 5.1Terminus 23 of 4
- GPT-5.3 CodexCodex CLI0 of 4
curve-quote-settle · python
One cell per agent and model, the same attempts in each, every trial recorded. A rate with no matrix behind it is an opinion about a model.
11 benchmark families we build sets for
A family is a suite built to order, and most of them draw on several of the categories below at once. Tell us which one you need and we will scope it against the capability you are trying to measure.
Commission a set- 01
LongHorizon FrontierSWE
Multi-hour software engineering on real repositories, where the work spans many files and the agent has to hold a plan across hundreds of turns.
- 02
LongHorizon SWE Marathon
The same shape taken further: sessions long enough that context management and recovery from a wrong turn are themselves what is being measured.
- 03
TB Frontier Science
Terminal-native scientific computing: data reduction, numerical pipelines, and whether a result reproduces rather than whether it looks plausible.
- 04
Terminal-Bench 3 and 4
The current Terminal-Bench generations, run under this platform's isolation and scoring rather than on an author's laptop.
- 05
Agent Safety
Utility and safety as two dimensions of one checkout, so an attempt that does the work and breaks a stated constraint fails rather than half-passing.
- 06
LongHorizon KnowledgeWork
Knowledge work rather than code, over long horizons: reconciliation against a written policy corpus, and reports somebody is accountable for.
- 07
Legacy Code Migration
Moving an old codebase forward, where the test suite is the only specification and the framework underneath it has changed since it was written.
- 08
Long-Horizon OSWorld 2
Desktop work through a GUI, where the state the agent has to reason about is on a screen rather than in a file it can read.
- 09
NL2Repo Samples
A whole repository from a natural-language brief, graded on what it does rather than on how it reads.
- 10
Multilingual SWE-Bench
The SWE-Bench shape outside Python, where the toolchain and the build system are part of the task rather than scenery.
- 11
CyberRL Samples
Offensive and defensive security work under isolation, graded on the outcome and on what the agent was not permitted to touch.
The 44 GDPval occupations, and we work in them too
GDPval names the occupations behind the nine sectors that each carry more than five percent of US GDP. We author and grade sets against them, and work of this kind is already in the catalogue above: statutory withholding and input-credit reconciliation is accountants’ work, and a payout run that has to balance is an analyst’s.
Ask about an occupationReal estate, rental and leasing
13.8% of GDP- Property, real estate and community association managers
- Counter and rental clerks
- Real estate sales agents
- Real estate brokers
- Concierges
Government
11.3% of GDP- Compliance officers
- Administrative services managers
- Child, family and school social workers
- First-line supervisors of police and detectives
- Recreation workers
Manufacturing
10.0% of GDP- First-line supervisors of production and operating workers
- Buyers and purchasing agents
- Shipping, receiving and inventory clerks
- Industrial engineers
- Mechanical engineers
Professional, scientific and technical services
8.1% of GDP- Software developers
- Lawyers
- Accountants and auditors
- Computer and information systems managers
- Project management specialists
Health care and social assistance
7.6% of GDP- Registered nurses
- First-line supervisors of office and administrative support workers
- Medical and health services managers
- Nurse practitioners
- Medical secretaries and administrative assistants
Finance and insurance
7.4% of GDP- Financial managers
- Customer service representatives
- Securities, commodities and financial services sales agents
- Personal financial advisors
- Financial and investment analysts
Retail trade
6.3% of GDP- General and operations managers
- First-line supervisors of retail sales workers
- Pharmacists
- Private detectives and investigators
Wholesale trade
5.8% of GDP- Sales representatives, wholesale and manufacturing, except technical and scientific products
- Sales managers
- Sales representatives, wholesale and manufacturing, technical and scientific products
- First-line supervisors of non-retail sales workers
- Order clerks
Information
5.4% of GDP- Producers and directors
- Editors
- News analysts, reporters and journalists
- Audio and video technicians
- Film and video editors
Sectors and occupations as published in GDPval (OpenAI, arXiv 2510.04374), with the GDP shares it reports. The list is theirs; the measurement on this site is ours, and no figure on this page comes from their benchmark.
What a set can be built in
Each category carries its own artifact contract, base image and resource ceiling, so a set starts from authoring rather than from plumbing. Where a category has finer grain it is listed beside it; most do not, and a set is then scoped against the category itself.
Data processing
in the catalogue- Entity matching
- Data contract
Software engineering
in the catalogue- Long context
- Tool specific
- API integration
- Database interaction
- UI building
- Issue resolution
- Marathon
Local cloud emulation
in the catalogueSecurity
in the catalogueCybersecurity
Finance and taxation
in the catalogueBuild and dependency management
Data science and reporting
Debugging
Games
Hardware
Data and infrastructure operations
Machine learning
- Tabular modelling
- Time series
Audio and media data engineering
Payroll and compliance
Scientific computing
- Terminal-Bench-Science
System administration
Tell us the capability you need measured, and we will scope a set against it
Asking for a task that is in the catalogue gets you the graded set, its images pinned by digest and the trial record behind the figures on its page. Asking for one that is not gets you a scope and a schedule, in the category or the family it belongs to.
Request a dataset