Gantry

Evaluation report, SynGrasp-1B

We fine-tuned the same policy twice on SynGrasp-1B, once on the release as published and once with the 583 episodes the release itself flags as failed grasps removed, and the edited run scored 33.6% against 28.3%, a gain of 5.3 points over 1,200 rollouts.

Fine-tuned on full
28.3%
Task success with all 17,540 episodes.
CI 25.9 to 30.9 · n=1,200
Fine-tuned on edited
33.6%
Task success with the 583 flagged episodes removed.
CI 31.0 to 36.3 · n=1,200
Lift
+5.3 pp
From training on less data.
95% range +1.6 to +8.9
Removed
3.3%
583 of 17,540 episodes, 48,369 of 1,449,589 frames.
81 minutes of 40.3 hours
Worst object class
27%
pick_up_propeller, 17 of 63 episodes flagged failed.
corpus rate 3.3% · 92 of 294 classes never fail

Success rate by task

fine-tuned on full SynGrasp-1B fine-tuned on edited SynGrasp-1B

A same-size random cut scored 28.6%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

pick_up_toast gains most, and pick_up_bracelet and pick_up_helicopter do not move at all. The pooled lift clears zero, most single tasks at 120 rollouts do not, and the chart says so.

What the edited version had removed

Measured on 17,540 episodes read from 12 of the 1,256 shards, every episode carries its own outcome flag in meta/episodes, and 583 of them, 3.3%, are flagged episode.success false and ship anyway. Nothing in the trajectory tells them apart, a flagged episode runs 82.97 frames on average against 82.63 for a kept one and the commanded gripper range is exactly 0.08 on all 17,540, so the flag is the only thing separating a grasp from a miss. The failures concentrate, pick_up_propeller loses 17 of 63 and pick_up_tennis_racket 14 of 53 against a corpus rate of 3.3% while 92 of the 294 object classes never fail once, and the per-class pattern repeats on two disjoint halves of the shard sample at a correlation of 0.59. The cut takes every flagged episode and no object class, so 46 of the 63 propeller episodes stay, because the thin and open frame shapes are the ones a grasping policy most needs and what it does not need is 17 recordings of the gripper closing on nothing.

Removedepisodesframes
flagged episode.success false, in the 7 object classes that fail above 20%958,063
flagged episode.success false, in the other 287 object classes47839,966
flagged and stopped at subtask 1 of 6, ten consecutive runs in one shard10340
Removed58348,369
Kept16,9571,401,220

The removed media

Measured on the release. Each clip is the window its episode occupies inside the packed front camera mp4, read from the from_timestamp column of meta/episodes, and every caption carries that episode's own outcome flag and the last subtask index its action stream reaches out of six.

728c6585/episode_000204removed
pick_up_propeller
113 frames · episode.success false · subtask 6 of 6 reached · this class fails 17 of 63
728c6585/episode_001441removed
pick_up_tennis_racket
115 frames · episode.success false · subtask 6 of 6 reached · this class fails 14 of 53
728c6585/episode_000651removed
pick_up_coat_hanger
93 frames · episode.success false · subtask 6 of 6 reached · this class fails 14 of 54
937e4b47/episode_000220removed
pick_up_bracelet
122 frames · episode.success false · subtask 6 of 6 reached · this class fails 14 of 59
fc410533/episode_001006removed
pick_up_sword
112 frames · episode.success false · subtask 6 of 6 reached · this class fails 10 of 47
22f9ce13/episode_000720removed
pick_up_box, then peach, grape and chinaware back to back
34 frames each · episode.success false · subtask 1 of 6 reached · ten consecutive runs stop this way
9bd1564c/episode_000646kept
pick_up_propeller, the same object class as the first clip
114 frames · episode.success true · subtask 6 of 6 reached · 46 of the 63 propeller episodes stay
fc410533/episode_001232kept
pick_up_coat_hanger, the same object class as the third clip
107 frames · episode.success true · subtask 6 of 6 reached · 40 of the 54 coat hanger episodes stay
SynGrasp-1B · 17,540 episodes measured of 4,920,219 released · GR00T N1.6 · 10 tasks × 120 rollouts × 2 runs Wilson 95% · cut = every episode the release flags episode.success false