Gantry

Evaluation report, InstructMove Dataset

We fine-tuned the same policy twice on the InstructMove release, once on all 500 episodes and once with the 300 dual arm episodes removed, and the edited run scored 24.3% against 19.7%, a gain of 4.6 points over 960 rollouts.

Fine-tuned on full
19.7%
Task success with all 500 episodes.
CI 17.3 to 22.3 · n=960
Fine-tuned on edited
24.3%
Task success with the 300 dual arm episodes removed.
CI 21.7 to 27.1 · n=960
Lift
+4.6 pp
From training on 200 episodes instead of 500.
95% range +0.9 to +8.3
Removed
60%
300 of 500 episodes, 44,562 of 78,980 frames.
24.8 minutes of 43.9 minutes
Still half of the action head
7 of 14
Dimensions holding at most two values from an episode's first frame to its last, in every dual arm episode.
160 left arm · 140 right arm · 0 of 8 on Franka

Success rate by task

fine-tuned on full InstructMove Dataset fine-tuned on edited InstructMove Dataset

A same-size random cut scored 19.9%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

The two biggest gains are both single object pick instructions on the dual arm robot, and one place instruction and one spatial pick instruction come out flat or below. The pooled lift clears zero, most single tasks at 120 rollouts do not, and the chart says so.

What the edited version had removed

Measured on the release, every episode was scored on how many of its own action dimensions hold at most two distinct values from its first frame to its last. That count is 7 of 14 in all 300 PiperX dual arm episodes, one entire arm, and 0 of 8 in all 200 Franka episodes. The still arm's total variation runs between 0.03 and 1.15% of the moving arm's, a swing of at most 0.014 rad over a whole episode, and which arm is still flips from episode to episode, 160 left and 140 right, so the constant half of the label can only be told apart from the video. The cut removes every episode carrying a dimension that never moves, which is the three PiperX subsets, 300 episodes and 44,562 of 78,980 frames.

Removedepisodesstill arm motion
pick by category, PiperX, one arm still from first frame to last1000.13%
pick by attribute, PiperX, one arm still from first frame to last1000.13%
place A to B, PiperX, one arm still from first frame to last1000.04%
Removed30044,562 frames
Kept20034,418 frames

The removed media

Measured on the release. Each clip is one episode read straight out of its subset zip by byte range, the dual arm ones from the static camera that sees both arms and the kept ones from the scene camera. The last two are kept on purpose, a scripted fixed duration trajectory is still a trajectory a policy can follow, while an arm that never leaves its start pose teaches the policy to hold seven of its fourteen outputs at a constant.

pick_category/episode_000018removed
Pick up the green-stem yellow curved banana.
106 frames · left arm 0.0039 rad against 5.7314 on the right · 7 of 14 dimensions still
pick_category/episode_000038removed
Pick up short rounded cheese sticks.
104 frames · right arm 0.0047 rad against 5.3187 on the left · 7 of 14 dimensions still
pick_attribute/episode_000081removed
Pick blue screwdriver up.
106 frames · left arm 0.0042 rad against 5.7450 on the right · 7 of 14 dimensions still
pick_attribute/episode_000035removed
Pick green comb up.
104 frames · right arm 0.0043 rad against 5.5203 on the left · 7 of 14 dimensions still
place_a2b_piperx/episode_000003removed
Grab hair clip with curved teeth and release it into round gray plate with raised edge.
237 frames · left arm 0.0039 rad against 13.8130 on the right · 7 of 14 dimensions still
place_a2b_piperx/episode_000057removed
Lift gray earbud case with hinge and drop into round gray plate with raised edge.
237 frames · right arm 0.0038 rad against 13.6567 on the left · 7 of 14 dimensions still
spatial_pick_franka/episode_000000kept
Grab the ac remote behind the measuring tape.
105 frames · 7.1695 rad across the single arm · 0 of 8 dimensions still
place_a2b_franka/episode_000001kept
Take long straight gray chopsticks and drop it in wide gray plate with blue rim.
239 frames · 13.7392 rad across the single arm · 0 of 8 dimensions still
InstructMove Dataset · 500 episodes · 78,980 frames · GR00T N1.6 · 8 tasks × 120 rollouts × 2 runs Wilson 95% · cut = any episode with an action dimension holding at most two values end to end