Gantry

Evaluation report, RAPT G1 OOD Benchmark

We fine-tuned the same detector twice, once on RAPT G1 OOD as released and once on an edited release with the 709 episodes shorter than the 200 step attribution window removed, and the edited one scored 5.2 points higher over 1,000 trials.

Fine-tuned on full
24.0%
Detection at the calibrated operating point with all 6,703 episodes.
CI 21.5 to 26.7 · n=1,000
Fine-tuned on edited
29.2%
Detection with the 709 episodes below the window removed.
CI 26.5 to 32.1 · n=1,000
Lift
+5.2 pp
From training on less data.
95% range +1.3 to +9.1
Removed
10.6%
709 of 6,703 episodes, every one shorter than 200 steps.
30.9 minutes of 53.7 hours
Length only AUROC
0.993
Log duration by itself separates the 39 anomalous real robot runs from the 11 nominal walks, and a threshold at the shortest nominal run catches 37 of the 39 at zero false positives.
0.58 for the same score on g1_velocity

Success rate by task

fine-tuned on full RAPT G1 OOD Benchmark fine-tuned on edited RAPT G1 OOD Benchmark

The ten rows are OOD categories the released test splits actually carry, ordered by how much of each the cut took. A same-size random cut scored 24.3%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

The categories that gain most are among the ones whose episodes end soonest. Pooled over the three simulation test splits, 29.8% of init_state episodes and 26.9% of obs_swap episodes run shorter than 200 steps, against 4.3% of noise and 3.9% of the nominal controls. The pooled lift clears zero, most single categories at 100 trials do not, and the chart says so.

What the edited version had removed

Measured on the release. Every episode's length was read exactly out of its own entry in the npz archive index, 6,703 episodes across the 14 archives, 9,670,865 logged steps at 50 Hz, 53.7 hours, without pulling the arrays themselves. The RAPT README says root cause attribution needs a 200 step history and that attribution over much shorter histories is unreliable, so the 709 episodes below that window were dropped, 10.6% of the episodes and 0.96% of the recorded time. 642 of those 709 come from the single task directory g1_velocity, which loses 20.5% of its episodes while g1_mimic_gangnam loses 1.4%. On the 50 run real robot split the 10 removed runs total 52 steps, 1.04 seconds between them, and 6 of them are a single timestep.

Removedepisodesmedian steps
g1_velocity nominal training and calibration episodes under the window462123
g1_velocity labeled test episodes under the window, 68 of them nominal controls180123
mimic task episodes under the window, 55 of them labeled anomalous test episodes57139
real robot runs under the window, 6 of them one timestep long101
Removed709123
Kept5,9941,500
RAPT G1 OOD · 6,703 episodes · RAPT detector · 10 OOD categories × 100 trials × 2 arms Wilson 95% · cut = episodes under the 200 step attribution window · release carries no camera stream