Gantry

Evaluation report, robocasa-PnPCounterToSink-MG_100_F_472

We fine-tuned the same policy twice on robocasa-PnPCounterToSink-MG_100_F_472, once on all 572 episodes as released and once with the 316 in which the gripper never holds the object for as long as 1.60 s removed, and the edited run scored 40.1% against 34.7%, a gain of 5.4 points over 700 rollouts.

Fine-tuned on full
34.7%
Task success trained on all 572 episodes.
CI 31.3 to 38.3 · n=700
Fine-tuned on edited
40.1%
Task success with the 316 episodes that never hold the object removed.
CI 36.6 to 43.8 · n=700
Lift
+5.4 pp
From training on less data.
95% range +0.4 to +10.5
Removed
55.2%
316 of 572 episodes, 3.07 of the 5.21 hours recorded.
every one of them runs the full 35.0 s horizon
Ends only at the horizon
82.5%
472 of 572 episodes are exactly 700 frames long and stop because the clock does.
all 472 carry a failure label · the other 100 carry no label at all

Success rate by task

fine-tuned on full robocasa-PnPCounterToSink-MG_100_F_472 fine-tuned on edited robocasa-PnPCounterToSink-MG_100_F_472

A same-size random cut scored 33.6%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

The boxed drink and milk instructions move most, and the cut emptied both hard, taking 10 of their 12 episodes and 9 of their 11. Garlic and mug sit flat, and they are the two the cut barely touched, losing 4 of 14 and 2 of 13. The pooled lift clears zero, most single instructions at 70 rollouts do not, and the chart says so.

What the edited version had removed

Measured on the release. Every episode reduces to one number, the longest unbroken stretch in which the gripper is commanded shut while the two finger joints stay more than 5 mm apart, which is the arm holding an object rather than closing on air, and all 100 episodes that end before the horizon hold for at least 1.60 s. 316 of the 572 never reach that, and all 316 are episodes that run the full 700 frame horizon, 35.0 s, and stop only because the clock does. The tail is a spike rather than a slope, 202 of the 316 peak under a single second and 3 hold nothing at all, against a median hold of 5.43 s in what stays, and it is legible in one channel only, because action dims 7 to 10 are numerically zero across all 375,092 frames and dim 11 is pinned at -1.0, leaving 5 of the 12 action dimensions carrying nothing. What stays is every episode where the arm actually held the object, including the 156 that held it and still ran the clock out, because a robot that grips the jug and cannot land it in the sink is a mistake the next model can see.

Removedepisodesmedian longest grasp
Holds nothing at any point, and in all 3 the gripper is never commanded shut in 35.0 s30.00 s
Longest hold under a second, the fingers shut and whatever was there is gone before the next second1990.70 s
Longest hold between 1.0 and 1.6 s, short of the weakest episode that finishes on its own1141.20 s
Removed3160.80 s
Kept2565.43 s

The removed media

Measured on the release. Each clip is the left agent view of one episode, the camera that has a file for all 572 of them, cut from the stretch around the grasp attempt, and the number in every caption is that episode's longest unbroken grasp, the seconds the gripper spent shut with its fingers held more than 5 mm apart.

episode_000369removed
pick the spray from the counter and place it in the sink
35.0 s recorded · longest grasp 0.00 s · the gripper is never commanded shut once · 1.11 m of gripper travel
episode_000495removed
pick the water bottle from the counter and place it in the sink
35.0 s recorded · longest grasp 0.75 s · 1 close attempt · 9 of the 13 water bottle episodes were removed
episode_000523removed
pick the squash from the counter and place it in the sink
35.0 s recorded · longest grasp 0.00 s · never commanded shut · 3.16 m of gripper travel
episode_000487removed
pick the wine from the counter and place it in the sink
35.0 s recorded · longest grasp 0.20 s · 4 close attempts · 13 of the 15 wine episodes were removed
episode_000504removed
pick the jug from the counter and place it in the sink
35.0 s recorded · longest grasp 0.65 s · 3 close attempts · 11 of the 18 jug episodes were removed
episode_000392removed
pick the carrot from the counter and place it in the sink
35.0 s recorded · longest grasp 0.10 s · shuts twice and holds for two frames · 1.31 m of gripper travel
episode_000000kept
pick the mango from the counter and place it in the sink
20.15 s recorded · longest grasp 7.75 s · ends 297 frames before the horizon, kept
episode_000351kept
pick the lemon from the counter and place it in the sink
35.0 s recorded · longest grasp 25.65 s · ran the clock out still holding the lemon, kept
robocasa-PnPCounterToSink-MG_100_F_472 · 572 episodes · 375,092 frames · GR00T N1.6 · 10 tasks × 70 rollouts × 2 arms Wilson 95% · cut = longest unbroken grasp under 1.60 s