Know what your robot data is worth before you train on it

Free · no account · Apache-2.0

No link? Upload a zip instead or see an example verdict

Gantry trains a policy on your dataset and tests it on real hardware. You get a report on whether the data is worth training on, and what to fix if it is not.

See what it has graded

What the bench has already graded

Lift a block, expert operators

robomimic / lift / ph

Passed

Clean expert demonstrations. Beat every one of its own shuffled controls, so the correspondence between what the camera saw and what the arm did carried real signal.

200 clips 20/20 controls beaten 13% error cut

Move a can, mixed operators

robomimic / can / mh

Passed

Skilled and unskilled operators in one pile, the messy case. Still beat the control on every derangement, which is the only claim the bench will make about it.

300 clips 20/20 controls beaten 20% error cut

Move a can, 75% of clips paired with the wrong video

damaged on purpose

Flagged

We broke this one deliberately. Three quarters of the clips carry another episode's actions, and the margin collapses to a coin flip instead of quietly passing.

200 clips 10/20 controls beaten 7% error cut

Move a can, every clip paired with the wrong video

damaged on purpose

Refused

Total damage, and the bench refuses instead of returning a number. Same pixels, same actions, no correspondence left, and nothing here is worth training on.

200 clips 0/20 controls beaten 2% error cut

65 runs on the record 50 clean datasets, zero false alarms every number ships with n and its interval

The evaluation rig, two robot arms at a shared table — Real hardware, not a simulator screenshot. This is where graded data earns its verdict.

The benchmark task family with workspace bounds annotated — Every task ships with its exact workspace bounds and tolerances, published up front.

We run the whole gauntlet

Submission to verdict, no configuration. Gates run in order of cost, and the cheap ones refuse a broken upload before anything expensive starts.

Gate 01 · Schema

Reads what you actually filmed

Counts, frame rates, licences, frozen streams, dropped frames. The bench inspects with the pipeline's own connectors, so what it sees is what training would see.

Inspectionlerobot pull

Episodes found312

Frame rate30.0 fps, stable

Licenceapache-2.0

Frozen camera streams4 episodes

Dropped frames0.2%

Gate 02 · Control

Builds a control out of your own data

Identical frames, identical actions, identical training budget. The only change is that each episode's actions get handed to a different episode's video.

Derangementseed 1 of 20

video ep_004→actions ep_117

video ep_005→actions ep_038

video ep_006→actions ep_291

video ep_007→actions ep_142

Gate 03 · Signal

Trains both and compares

Your data has to beat the shuffle. Whatever fine-tuning-in-general buys, the control arm buys too, so only correspondence can explain a win.

Held-out clipsmargin

Your dataset13/13 won

Shuffled controlbaseline

Prediction errorcut 74%

Exact sign testp = 0.000

Gate 04 · Verdict

Tells you what to fix

Pass with the margin and its interval, refuse with the failing gate and a prescription, or abstain with the reason shown. Never a bare score.

Prescription4 items

1Refilm 4 episodes where the wrist camera froze mid-grasp

2Resync ep_088 to ep_113, actions lead the video by 6 frames

3Drop 11 episodes that never complete the task

4Keep the rest. 288 episodes pass unchanged

Three things a loss curve cannot tell you

50 / 50

Clean data never false-alarmed

Fifty intact datasets across three task families, every one passed. A screen that cries wolf on good data is worse than no screen, so we measured that first.

±1 frame

Timing damage is invisible to statistics

Delay every action by up to 20 frames and no dependence score flinches. So a separate gate hunts the offset directly and recovers it to the frame, in 12 seconds.

1 min vs 17 h

Cheap enough to run on everything

The probe lands on the same verdicts as a method that burns 17 GPU hours on VAEs and kNN estimators. Ours runs on a laptop while you wait.

Every run we have on the record

Including the ones we damaged on purpose to see what escapes the grade.