Overview

The experiment as a pipeline, and where it stands. Every number carries its sample size and interval; anything the bench could not measure is shown as unmeasured rather than as zero.

Needs attention

The strongest findings from the last diagnosis. Everything else is under Diagnosis.

Latest runs

Runs

Closed-loop evaluations on expert-screened scenes. Expand a row for the stage ladder and execution detail.

Compare

Two arms on the same scenes, paired. Most scenes agree and carry no information; the test runs on the disagreements only.

Pick two runs that carry per-scene records.

Diagnosis

Findings from every feedback module, grouped by check. One row per finding; the cohorts it applies to are chips on the row. Expand for the evidence and the prescription.

Could not tell you

Modules that declined to answer, with the reason. Shown rather than dropped.

Datasets

What the bench has read or built, and what each declared. Uploads are inspected with the pipeline's own connectors; nothing is trained or modified.

Drop a LeRobot dataset (.zip) or click to choose

Contributor split

Two cohorts cut from one corpus on a measured property, the fraction of frames in which both hands were moving.

On the bench

Tutorial, your first verdict

From a dataset on your disk to a scored verdict, in five steps. Nothing you upload is trained on or modified during inspection.

1. Package the dataset

The bench reads LeRobot datasets. Zip the dataset folder as exported by your recording pipeline, the folder that holds meta/, data/ and videos/. Do not re-arrange files inside it.

2. Drop it on the bench

Go to Datasets and drop the zip on the upload target. Inspection starts immediately and uses the pipeline's own connectors, so what the bench sees is exactly what training would see.

3. Read the inspection

The first report is cheap and instant: episode and frame counts, frame rate, schema, licences, and per-episode screens. Problems appear as coded findings with plain-language messages, not as silent failures.

4. Let the gauntlet run

Behind the inspection, the gates run in order of cost. The core measurement is the shuffled control: a probe is fitted to your data, and again to your data with every episode's actions handed to a different episode's frames. Your data has to beat that second arm, beating "training moved the number" is not enough.

5. Read the verdict

Every number carries its sample size and interval. A gate that cannot answer abstains and says why rather than guessing. Findings that call for action carry a prescription, what to fix, what to refilm, under Diagnosis.

How it works

The measurement, and why it is built this way.

The problem with before-and-after

Fine-tuning a large model on anything moves it. Loss falls, behaviour changes, and a before-and-after table shows a difference, for every dataset, including worthless ones. A score that comes back positive for everyone is not a measurement.

The shuffled control

So the bench builds the baseline out of your own data: identical frames, identical actions, identical training budget, and the actions belong to a different episode than the frames do. Whatever fine-tuning-in-general buys, that arm buys too. Beating it is the only thing that says the correspondence between what was seen and what was done carried information.

One gate per failure mode

No single check can see every defect, we measured this. A dependence screen is provably blind to a sync offset between video and actions, because a time shift preserves dependence; a cheap cross-correlation gate recovers the same offset to the frame. So the bench runs a ladder of narrow gates, each catching what the others cannot, in order of cost.

Verdict semantics

Three honest outcomes, never blended: pass with the margin and its interval; refuse with the failing gate and a prescription; and abstain when a gate could not answer, with the reason shown rather than dropped. Rates come with n and Wilson 95% intervals, everywhere.

How-to guides

Short, task-shaped answers. Expand the one you need.

Submit a dataset

Zip the LeRobot dataset folder and drop it on Datasets. Keep the original folder layout; the bench reads meta/info.json for schema and fps. Large uploads are inspected streaming, you can leave the tab.

Read a verdict

Start at the margin: how much better your data predicted actions than its own shuffled control, with its interval. Then the per-gate rows: green passed, red refused with a prescription, grey abstained with a reason. A refused gate is a finding about the data, not a rejection of you, the prescription says what to change, and resubmission is free.

Fix a refused dataset

Each refusal carries a code and a prescription under Diagnosis. The common ones: correspondence failures (check that actions and video come from the same take and the same clock), sync offsets (re-mux with the estimated shift the sync gate reports), and frozen streams (a tracker that lost the arm reads as a solid block of identical values). Fix, re-zip, resubmit.

Compare two runs fairly

Use Compare. Runs pair scene by scene; scenes where both arms agree carry no information, so the test runs on the disagreements only. Runs without per-scene records cannot be compared and say so.

Submissions

Everything the bench has been handed, newest first, with where it stands.

New submission? Drop a LeRobot zip on Datasets, it appears here once inspected.

Reports

The written record, as committed to the repository.