Task pipeline, specification

A task is not data until it has been graded

We build containerised tasks for frontier labs and put every one through six gates before it counts as a datapoint. The reference solution has to work. An agent that does nothing has to fail. The pass rate has to land in a band. And a person has to have read the transcript.

Request a dataset Sign up to write tasks

An approval stamp inked onto a printed specification sheet — gate 2 · oraclereward 1.00 · 5 of 5

6 gates · 6 intake stages

Stages 1 to 5 execute nothing. intake-no-exec.test.ts proves it.

17
task categories: 6
gates before a score counts: 6
intake stages before a byte runs

Coverage

Seventeen capabilities, measured in containers

Each category carries its own artifact contract, base image and resource ceiling, so a scientific-computing task is not graded by the rules that suit a payroll one.

01
Software engineering
Evaluates whether an agent can change a real codebase and keep its tests passing.
02
Debugging
Measures whether an agent can localise a fault from a failing run and repair it.
03
Security
Evaluates whether an agent finds and fixes a vulnerability without weakening the rest.
04
Cybersecurity
Measures a defensive source audit end to end: the service is patched to match its own documentation and every finding is reported, without the service being replaced or disabled.
05
System administration
Measures command-line competence against a machine that must end in a given state.
06
Data and infrastructure operations
Evaluates whether an agent can operate running infrastructure rather than describe it.
07
Build and dependency management
Measures whether an agent can make a broken build resolve and compile again.
08
Data processing
Evaluates transformation of real files into an exactly specified output.
09
Scientific computing
Measures numerical work where the answer is checked to a tolerance, not by eye.
10
Machine learning
Evaluates whether an agent can train, evaluate and ship a model that clears a bar.
11
Data science and reporting
Measures whether an analyst deliverable still holds on data it has never seen: the program is re-fitted on a hidden world and scored to a tolerance, with the governing document buried among superseded ones.
12
Payroll and compliance
Measures arithmetic and rule-following where one wrong cell fails the task.
13
Finance and taxation
Measures whether an agent can apply statutory arithmetic that is dated, jurisdictional and held somewhere other than the repository.
14
Audio and media data engineering
Evaluates handling of binary media formats against a verifiable artifact.
15
Games
Measures planning over long horizons where a score, not a diff, is the outcome.
16
Hardware
Evaluates work against physical or device-level constraints the software categories do not name.
17
Local cloud emulation
Evaluates an agent driving a local cloud-shaped control plane, not a description of one.

Marketplace

The tasks, and the model each one already beat

Each of these was run under gate 5 before it was listed, and carries the pass rate that round measured. Every one opens onto the per-model breakdown behind its figure.

The instruction, the tests and the reference solution stay with us. What a licence buys is the graded set and the trial record behind these figures.

Open the catalogue Request a dataset

Built to order

These categories are defined, with their own artifact contract, base image and resource ceiling, so a set in one of them starts from authoring rather than from plumbing. Tell us the capability you need measured.

Cybersecurity
Build and dependency management
Data science and reporting
Debugging
Games
Hardware
Data and infrastructure operations
Machine learning
Audio and media data engineering
Payroll and compliance
Scientific computing
System administration

Separately from the categories, 11 benchmark families are listed in the catalogue, each with the capability it measures.

Request a set See the 11 families

Scoring

A score only counts if the task was hard enough

Difficulty is measured on the model matrix before a task is admitted — not estimated by whoever wrote it. Land outside the band and the task is rejected, however interesting it looks.

Measured pass rate across the model matrixA band, not a maximum

Too hard

Admitted

Rejected as too easy

0%10%60%80%100%

Between 10% and 60% a task is admitted. Above 80% it is rejected: a benchmark a model passes on the first try measured nothing. Below 10% it is usually the harness that is broken, not the model.

One measurement

Every number on this platform came out of a container

Not from a rubric a model filled in, and not from a sample somebody eyeballed. Four steps, and all four leave something you can open afterwards.

01
An isolated container
Built rootless on a dedicated pool and pinned by digest. Network egress is denied by default, so a trial cannot reach anything it was not granted.
02
The agent attempts it
It gets the instruction and a working directory. The test suite is not in its image — a submission that copies tests into the agent is refused before it ever runs.
03
A verifier scores it
A separate image runs the checks and writes a reward to /logs/verifier/reward.txt. The number comes out of the container; no model judges another model.
04
The trial is kept
Reward, tokens, cost, wall time and the whole trajectory are stored as a row. A score you cannot open is a score you cannot check.

In the matrix

Every trial runs an agent against a named model from the registry, and which models those are is read from the registry rather than written into this page — a model is eligible only while it is marked ready, and retired ones are kept so the results they already produced stay readable. The models a difficulty probe runs against today.

The standard

A benchmark a model passes on the first try measured nothing. The useful task is the one that fails honestly — and proving that takes more work than writing it.

Which is why nothing here is published on an author's word. The pass rate is measured, the reward is written by a verifier inside the container, and a person who did not write the task has to read it fail before it counts.

Intake

Nothing runs until the sixth stage

An uploaded bundle is a stranger’s code. The first five stages read it, measure it and refuse it without executing a single byte — and that is a property we test, not a promise we make.

intake-no-exec.test.ts

Walks the import graph of stages 1–5 and fails if anything on it can reach child_process, vm, eval or a dynamic import.

Two reviewers reading a failing trajectory together at one screen — Gate 6 — human review2 trajectories, minimum

01
Receive
Oversized bundles, and git URLs pointing at private or metadata addresses.
02
Unpack
Path traversal, symlink escape, setuid entries, zip bombs, duplicate members.
03
Structure
A bundle missing instruction.md, task.toml, tests/test.sh or a Dockerfile.
04
Normalise
A verifier the agent could reach, and resources above the tier.
05
Scan
Docker sockets, credential reads, remote code loads, tests leaked into the agent image.
06
Build
first execution
Anything that will not build rootless, behind an allowlist, pinned by digest.

A row of server racks receding down a data hall, status lights along the rack faces

Where it runs

Every trial is a container on an isolated pool, built rootless, pinned by digest, with the network denied by default.

A dark panel showing a dense grid of evaluation cells, a scattered few lit in blue — For labsModel matrix — every runnable agent and model, a fixed number of attempts

For labs

Measure your model where it actually breaks

You get the score and everything behind it: the bundle, the image digests, every trial that ran, and the verdict at each gate. Not a CSV of prompts and a number you have to take on trust.

Scored in containers: Reward is written by a verifier inside the sandbox. There is no model judging another model anywhere in the scoring path.
Reproducible across versions: Agent and verifier images are built rootless and pinned by sha256, so re-running a score next quarter runs the same bytes.
Hard by measurement: Every task carries the pass rate that admitted it, measured across 120 trials on the model matrix rather than asserted by its author.
Seventeen capabilities: Each category has its own artifact contract, base image and resource ceiling, so a result is attributable to a capability.

Benchmark your model

A long fanfold printout of dense pass and fail marks on a pale bench, with a steel ruler — For authorsTrial log — one task, every attempt

For authors

Write the tasks the frontier still fails

A task only earns its place if models fail it honestly. The pipeline measures that for you and says exactly why a submission did not qualify, with the file and the line.

Difficulty is measured for you: Gate 5 runs your task across the matrix and reports the pass rate. You do not have to guess whether it is hard enough.
Findings, not a rejection: Each refusal carries a remediation, so a failed check names the fix rather than only the failure.
A real reviewer: Gate 6 requires two failing trajectories to be opened by someone who is not you, recorded as access events.
Paid on accept: Acceptance enqueues the payout in the same transaction as the verdict, so the two cannot disagree.

Apply to write

A task is not data until it has been graded

Seventeen capabilities, measured in containers

Software engineering

Debugging

Security

Cybersecurity

System administration

Data and infrastructure operations

Build and dependency management

Data processing

Scientific computing

Machine learning

Data science and reporting

Payroll and compliance

Finance and taxation

Audio and media data engineering

Games

Hardware

Local cloud emulation

The tasks, and the model each one already beat

Merge a coding agent's queued messages into one turn

Plan which MCP tools a turn can offer

Rebuild a settlement ledger against withholding mints

Report every file a reference library holds

Reconcile an inbox against responses that arrive out of order

Recover a session's context figure from its transcript

Decide what a command string would actually run

Reconcile a month's input tax credit

Decide which test results are evidence about production

Decide which imported fields were altered on the way in

Report who holds a resource's data after a chain of hand-offs

Take a leaver out of the team tables

Compose SSH command batches from a maintenance plan

Close a replay gap in a TOTP verify path

Find the money in a payout run that reported clean

Withhold tax against a schedule only a service holds

A score only counts if the task was hard enough

Every number on this platform came out of a container

An isolated container

The agent attempts it

A verifier scores it

The trial is kept

Nothing runs until the sixth stage

Receive

Unpack

Structure

Normalise

Scan

Build

Measure your model where it actually breaks

Write the tasks the frontier still fails