Gantry

Evaluation report, PlayWorld human demonstration data

We fine-tuned the same policy twice on the PlayWorld human demonstration release, once on all 925 episodes as shipped and once with the 140 whose instruction never names the manipulation or whose end effector sits still for a fifth of the recording removed, and the edited run scored 25.3% against 20.5%, a gain of 4.8 points over 750 rollouts.

Fine-tuned on full
20.5%
Task success trained on all 925 episodes.
CI 17.8 to 23.6 · n=750
Fine-tuned on edited
25.3%
Task success with the 140 removed episodes gone.
CI 22.4 to 28.6 · n=750
Lift
+4.8 pp
From training on less data.
95% range +0.5 to +9.0
Removed
15.1%
140 of 925 episodes, 35.8 of the 214.8 minutes recorded.
10,752 of 64,454 stored frames
Labelled a success
925 of 925
The per-episode success field reads 1 on every episode in all six version and split directories, so nothing in the release marks a bad demonstration.
distinct values in that field · 1

Success rate by task

fine-tuned on full PlayWorld human demonstration data fine-tuned on edited PlayWorld human demonstration data

A same-size random cut scored 20.7%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

Move object into or out of container holds 726 of the 925 episodes and 35 of the removals, and it moves furthest. Open or close curtain lost one episode of forty two and does not move, and move lid on or off of container lost none and gives two rollouts back. The pooled lift clears zero, most single tasks at 150 rollouts do not, and the chart says so.

What the edited version had removed

Measured on the full release, all 925 episodes, not a sample. Every episode ships one annotation JSON holding an instruction string, a success flag and the end effector pose at each stored video frame, and the success flag reads 1 on all 925, in all six version and split directories, so it cannot separate a good demonstration from a bad one and the filter has to come out of the trajectory instead. Reading the pose, the share of an episode's steps on which the end effector translates under a millimetre, turns under a thousandth of a radian and leaves the gripper where it was runs from 0 to 50 percent with a median of 9.8, and covers 7,256 of the 63,529 frame to frame steps in the release, 24.2 of the 214.8 minutes. The cut takes the 102 episodes whose instruction is a DROID collection prompt such as Do anything you like that takes multiple steps to complete rather than a description of the manipulation, and the 38 more that hold still on a fifth or more of their frames, 140 episodes and 35.8 minutes together. What stays is every episode where the arm fumbles, overshoots or re-grasps, because a mistake the next model can see is training signal and a still arm is not.

Removedepisodesshare of the release
Instruction is a collection prompt, not a description of the manipulation9410.2%
End effector still on a fifth or more of the episode's stored frames384.1%
Both faults in the same episode80.9%
Removed14015.1%
Kept78584.9%

The removed media

Measured on the release. Each clip is the exterior_1 camera of one episode, stored at 320 by 192 and 5 frames a second, so one video frame is one row of the pose stream and the two line up exactly. The number in every caption is the share of that episode's steps on which the end effector does not move, and where it matters the longest unbroken run of them.

v1_final . train . 14removed
Move object into or out of container
113 frames, 22.6 s · still on 56 of 112 steps, 50.0% · an 8.4 s unbroken freeze
v1_final . train . 15removed
Move object into or out of container
139 frames, 27.8 s · still on 59 of 138 steps, 42.8% · a 10.6 s freeze, the longest run in the release
v0_final . train . 158removed
Do any three tasks consecutively.
74 frames, 14.8 s · still on 29 of 73 steps, 39.7% · both faults, the instruction names no object
v2_final . train . 30removed
Do any task, and then reset the scene.
130 frames, 26.0 s · still on 43 of 129 steps, 33.3% · both faults, the reset is inside the demonstration
v0_final . val . 69removed
Do anything you like that takes multiple steps to complete.
96 frames, 19.2 s · still on 26 of 95 steps, 27.4% · that sentence is the whole instruction
v0_final . train . 33removed
Do any two tasks consecutively.
89 frames, 17.8 s · still on 14 of 88 steps, 15.9% · removed on the instruction, not on the motion
v1_final . train . 282kept
Move object to a new position and orientation
116 frames, 23.2 s · still on 2 of 115 steps, 1.7% · instruction names the manipulation, kept
v0_final . train . 168kept
Move object into or out of container
97 frames, 19.4 s · moves on 93 of its 96 steps, still 3.1% · kept
PlayWorld human demonstration data · 925 episodes · 214.8 minutes · GR00T N1.6 · 5 tasks × 150 rollouts × 2 runs Wilson 95% · cut = a collection prompt for an instruction, or the end effector still on a fifth or more of the stored frames