Gantry

Evaluation report, PhysicalAI-Robotics-GR00T-Teleop-Sim

We fine-tuned the same policy twice on PhysicalAI-Robotics-GR00T-Teleop-Sim, once on all 24,000 takes as released and once with the 274 takes whose teleoperator froze for five seconds or more removed, and the edited run scored 31.6% against 26.4%, a gain of 5.2 points over 1,200 rollouts.

Fine-tuned on full
26.4%
Task success trained on all 24,000 takes.
CI 24.0 to 29.0 · n=1,200
Fine-tuned on edited
31.6%
Task success with the 274 frozen takes removed.
CI 29.0 to 34.3 · n=1,200
Lift
+5.2 pp
From training on less data.
95% range +1.5 to +8.8
Removed
3.4%
197,368 of 5,820,277 frames, 274 takes, 2.74 of the 80.84 hours recorded.
0.81 of those hours the command never moved
Reached by the declared split
10%
All 24 directories declare 1,000 episodes and ship a declared train split that stops at episode 100.
2,400 of 24,000 takes · 9.87% of frames

Success rate by task

fine-tuned on full PhysicalAI-Robotics-GR00T-Teleop-Sim fine-tuned on edited PhysicalAI-Robotics-GR00T-Teleop-Sim

A same-size random cut scored 26.1%, so the gain is from which takes were removed, not from removing takes.

Gain per task

The wine cabinet and the plate to pan directories move most, and they are second and fifth by how many takes the cut took out of them. The cup drawer lost 29 takes and did not move, and the plate to cardboard box lost 12 and did not move either. The pooled lift clears zero, most single directories at 120 rollouts do not, and the chart says so.

What the edited version had removed

Measured on the release. Every take reduces to one number, the seconds its 44 joint command vector sits still, where a frame counts as frozen when the largest absolute change on any of the 44 joints since the previous frame is under 1e-2, read at the declared 20 fps. Ranking all 24,000 takes by length against their own directory median costs nothing because meta/episodes.jsonl carries the length of every one, so we pulled the action column for the 680 takes that run at twice their directory median or longer and for a random 250 of the remaining 23,320 as a control, 930 takes measured in all. Those 680 long takes are frozen for 18.1% of their recorded time against 8.0% for the control, and the 274 of them that hold still for five seconds or more carry 2.74 hours of recorded robot time of which 0.81 hours is a motionless command. The cut lands on the six pick and place then close directories, a quarter of the release that gives up 157 of the 274, and hardest on PnPMilkToMicrowaveClose at 55 of its 1,000. Something else fell out of the same pull, 63 pairs among those 680 match exactly on length, frozen frame count and longest freeze, 126 takes in all, and the seven pairs we opened frame by frame are identical on every one of the 44 joints. What stays is the 406 long takes that keep moving, one of them running 4.49 times its directory median with 1.45 seconds frozen, because a take that runs long on a grasp that went wrong is something the next model can learn from and a take that runs long because nobody was driving is not, and only 8 of the 250 control takes would have cleared the same five second bar, so the shorter takes are left alone.

Removedtakesshare of that directory
PnPMilkToMicrowaveClose, the worst single directory555.5%
PnPWineToCabinetClose, PnPCupToDrawerClose, PnPCanToDrawerClose832.8%
PosttrainPnPNovelFromPlacematToTieredshelfSplitA and PosttrainPnPNovelFromPlateToPanSplitA522.6%
the other 18 directories840.5%
Removed2741.1%
Kept23,72698.9%

The removed media

Measured on the release. Each clip is the ego_view camera of one take, the only camera the release ships, cut at the longest stretch where the command was frozen. The number in every caption is that take's frozen seconds, recorded robot time where none of the 44 joints moved.

PnPWineToCabinetClose . episode_000100removed
pick up the wine, place it into the cabinet and close the cabinet
1,790 frames · 89.50 s recorded · 52.80 s frozen · 4.51x its directory median
PosttrainPnPNovelFromTrayToTieredshelfSplitA . episode_000827removed
pick the lemon from the tray and place it in the tiered shelf
1,087 frames · 54.35 s recorded · 32.10 s frozen · 59.1% of its frames
PosttrainPnPNovelFromPlateToPanSplitA . episode_000110removed
pick the cupcake from the plate and place it in the pan
839 frames · 41.95 s recorded · 30.10 s frozen · one unbroken freeze runs 21.55 s
PosttrainPnPNovelFromPlacematToTieredshelfSplitA . episode_000976removed
pick the croissant from the placemat and place it in the tiered shelf
899 frames · 44.95 s recorded · 29.55 s frozen · 65.7% of its frames
PnPMilkToMicrowaveClose . episode_000253removed
pick up the milk, place it into the microwave and close the microwave
997 frames · 49.85 s recorded · 22.65 s frozen · identical on all 44 joints to episode_000549
PnPCanToDrawerClose . episode_000948removed
pick up the can, place it into the drawer and close the drawer
740 frames · 37.00 s recorded · 21.30 s frozen · 57.6% of its frames
PosttrainPnPNovelFromPlateToPlateSplitA . episode_000323kept
pick the squash from the plate and place it in the plate, kept
907 frames · 45.35 s recorded · 1.45 s frozen · 4.49x its directory median and still kept
PnPWineToCabinetClose . episode_000319kept
pick up the wine, place it into the cabinet and close the cabinet, kept
298 frames · 14.90 s recorded · 0.05 s frozen · same instruction as the first clip, 0.75x the median
PhysicalAI-Robotics-GR00T-Teleop-Sim · 24 directories · 24,000 takes · 5,820,277 frames · GR00T N1.6 · 10 tasks × 120 rollouts × 2 runs Wilson 95% · cut = five or more frozen seconds in a take at twice its directory median