Seventeen capabilities, measured in containers
Every category carries its own artifact contract, base image and resource ceiling, so a scientific-computing result is not graded by the rules that suit a payroll one, and a score is attributable to a capability rather than to an average.
- 17
- task categories
- 6
- gates before a score counts
- 12
- trials per difficulty gate
- 114
- refusals before anything runs
Each with its own artifact contract, base image and size ceiling rather than one global guess.
Build, oracle, do-nothing, new-seed, difficulty, then a person. Five are automatic.
Read from gate-5 policy across 11 rows, not typed into this page. A task's admitted pass rate is measured, not asserted.
83 of them block a submission outright. Every one is a row in validation_rule with a stage, a severity and a remediation, counted from the platform rather than typed into this page. The other 31 are recorded as a warning without stopping anything.
The agentic set
Every row of `category`, with what each one evaluates. The verb is the point: these are things that measure a model, not things an author writes.
- Software engineering
- Evaluates whether an agent can change a real codebase and keep its tests passing.
- Debugging
- Measures whether an agent can localise a fault from a failing run and repair it.
- Security
- Evaluates whether an agent finds and fixes a vulnerability without weakening the rest.
- Cybersecurity
- Measures a defensive source audit end to end: the service is patched to match its own documentation and every finding is reported, without the service being replaced or disabled.
- System administration
- Measures command-line competence against a machine that must end in a given state.
- Data and infrastructure operations
- Evaluates whether an agent can operate running infrastructure rather than describe it.
- Build and dependency management
- Measures whether an agent can make a broken build resolve and compile again.
- Data processing
- Evaluates transformation of real files into an exactly specified output.
- Scientific computing
- Measures numerical work where the answer is checked to a tolerance, not by eye.
- Machine learning
- Evaluates whether an agent can train, evaluate and ship a model that clears a bar.
- Data science and reporting
- Measures whether an analyst deliverable still holds on data it has never seen: the program is re-fitted on a hidden world and scored to a tolerance, with the governing document buried among superseded ones.
- Payroll and compliance
- Measures arithmetic and rule-following where one wrong cell fails the task.
- Finance and taxation
- Measures whether an agent can apply statutory arithmetic that is dated, jurisdictional and held somewhere other than the repository.
- Audio and media data engineering
- Evaluates handling of binary media formats against a verifiable artifact.
- Games
- Measures planning over long horizons where a score, not a diff, is the outcome.
- Hardware
- Evaluates work against physical or device-level constraints the software categories do not name.
- Local cloud emulation
- Evaluates an agent driving a local cloud-shaped control plane, not a description of one.
Difficulty is measured, not asserted
A task is admitted only when its measured pass rate lands between 10% and 60% across the model matrix. Above 80% it is rejected as too easy; below the floor it is usually the harness that is broken rather than the model.
- Where the number comes from
- 12 trials on the model matrix — run before the task is admitted. Nobody is asked to estimate it. The size is gate-5 policy, read from the platform across 11 policy rows, not a figure typed into this page.
- Why there is a ceiling at all
- A benchmark a model passes on the first try measured nothing. The band is the product: it is what makes the set hard by construction rather than by assertion.
- Errored trials
- A sandbox timeout or a preemption is a fact about our fleet, not about the model, so it is excluded from the denominator rather than counted as a failure.
Leaderboards, and why none is published yet
The machinery is real and the boards are empty, which is worth saying plainly rather than filling with a placeholder.
- What a board is
- A rollup of trials: pass@k per agent-and-model cell for one dataset version, derived by a job rather than written by hand. A board is one value of k and never a mixture of several.
- What it carries
- Every contributing run, so a published figure can be opened. A pass rate with no way to reach the trials behind it is a number nobody can audit.
- Why this page shows none
- No dataset version has been published yet, so there is nothing to rank. A ranking invented for the look of it would discredit every real figure on this site.