Gantry

Evaluation report, v30_apple_storage

We fine-tuned the same policy twice on v30_apple_storage, once on the release as published and once with every frame recorded after the apple is already in the plate taken out, and the edited version scored 5.3 points higher across 600 graded rollouts.

Fine-tuned on full
20.3%
Stage success with all 899 frames.
CI 17.3 to 23.7 · n=600
Fine-tuned on edited
25.7%
Stage success with the post-task frames taken out.
CI 22.3 to 29.3 · n=600
Lift
+5.3 pp
From training on 58% fewer frames.
95% range +0.6 to +10.1
Removed
58.4%
525 of the 899 frames are recorded after the apple is already in the plate.
17.50 of the 29.97 seconds
Header against release
16.7x
meta/info.json declares 14,983 frames and a train split of 0 to 50. The three episodes hold 899.
14,084 declared frames are not in the repo

Success rate by task

fine-tuned on full v30_apple_storage fine-tuned on edited v30_apple_storage

A same-size random frame cut scored 20.6%, so the gain is from which frames were removed, not from removing frames.

Gain per task

The release stage gains most, and it is the stage the removed frames sit immediately after, so the policy stops spending capacity on what the arm does once the apple is already down. Carry is flat. Only release clears zero on its own at 150 rollouts, the pooled lift clears it, and the chart says so.

What the edited version had removed

Measured on the whole release, all three episodes and all 899 frames, from 239 KB of metadata and the one 23.8 MB video rather than a sample. The gripper command sits at one of two values on 884 of the 899 frames, 130.0 degrees open and -5.0 closed, so the frame where the apple is let go over the plate is exact rather than a judgement, and it lands at 4.60 s, 4.13 s and 3.73 s inside three takes that each run a flat 10.0 seconds. 525 of the 899 frames, 58.4% of the release, are recorded after the task is already finished, 17.50 seconds out of 29.97, and every take loses more than half of itself at 53.8%, 58.7% and 62.7%. The arm is still retracting through part of that, so the cut drops 58.4% of the frames but only 40.6% of the commanded arm travel, 396 degrees of 974.

Removedframessecondsshare of the release
the arm retracting once the apple is on the plate34811.6038.7%
the same retract with the head turned off the workspace411.374.6%
the arm parked, no commanded joint moving for a second or more1364.5315.1%
Removed52517.5058.4%
Kept37412.4741.6%

The removed media

Measured on the release. The first three clips each open on the exact frame the gripper reopens over the plate and run to the end of that take, so everything in them is what the cut removes, and the next three are the moments inside that tail worth naming on their own. The 189 frames where the commanded gripper sits 67 to 126 degrees away from the measured gripper are kept, not cut, because the apple is holding the jaws open and a policy can see it in the hand, and on the other eighteen channels the same measure never exceeds 15.1 degrees.

episode 0 tailremoved
The gripper opens over the plate at 4.60 s and the recorder runs another 5.37 s.
299 frames · 161 removed · 53.8% of the take
episode 1 tailremoved
Released at 4.13 s of 10.00, and the head leaves the table 1.1 s after that.
300 frames · 176 removed · 58.7% of the take
episode 2 tailremoved
The longest tail in the release, released at 3.73 s of a 10.00 s take.
300 frames · 188 removed · 62.7% of the take
episode 1 at 15.23 sremoved
The head turns off the table and settles on a person across the room while the arm is still being commanded.
41 frames · 1.37 s · head commanded 31 degrees off the pose it held at release
episode 0 at 7.00 sremoved
Parked with the apple already down, 41 straight frames with no commanded joint moving 0.75 degrees.
41 frames · 1.37 s · inside the removed tail
episode 2 at 28.20 sremoved
The take runs out its ten seconds on 53 frames of nothing.
53 frames · 1.77 s · last commanded motion at 8.23 s of 10.00
episode 2 keptkept
Everything the cut keeps in episode 2, reach, close on the apple, carry, release.
112 frames · 3.73 s · 146 of the take's 266 degrees of arm travel
episode 0 graspkept
The grasp kept on purpose, gripper commanded to -5.0 degrees while the apple holds the jaws at 79.0.
78 frames · commanded 67 to 116 degrees from measured · kept
v30_apple_storage · 3 episodes · 899 frames · 4 stages × 150 rollouts × 2 arms Wilson 95% · cut = every frame after the gripper reopens over the plate