Evaluation report, cpaxton/pickup_pink_cup
We fine-tuned the same policy twice on cpaxton/pickup_pink_cup, once on the release as published and once with the 2 takes removed whose gripper command never covers the open to closed span the other 18 do, and the edited version scored 5.2 points higher across 600 graded rollouts.
Success rate by task
A same-size random cut of 2 takes scored 19.8%, so the gain is from which takes were removed, not from removing takes.
Gain per task
The pooled gain clears zero with a low end of +0.5 points. Two of the six stages come out flat on their own at this trial count, and the chart says so.
What the edited version had removed
Measured on the release. All 20 takes carry the same task string, the same operator field and the same environment field and were recorded inside one hour on 15 August 2024, so the only thing separating them is what the arm did. The per-episode number is gripper command travel, the distance the stretch_gripper channel covers inside a take, and it splits the release cleanly, 18 takes between 0.8199 and 0.8701 and two below 0.6200. Episode 7 holds the gripper at 0.6000, its fully open value, on all 125 frames, while the lift joint runs 0.5714 down to 0.2634 and back up to 0.5386, a travel of 0.3080 that sits inside the 0.2441 to 0.4312 the other takes cover, so the arm completed a pick with nothing in the hand. Episode 3 never opens past 0.3634 against the 0.5945 to 0.6000 the other 19 reach, so its whole approach happens with the hand part way shut. The cut is those two, 260 frames of 2,587. The three takes running 1.5 to 1.8 times the median 126 frames are kept, and so is the take whose gripper is closed on only 13 of its 127 frames, because a slow or clumsy demonstration that still closes on the cup is something a policy can learn from and a pick that grasps nothing is not. Two more channels carry nothing at all, base_y_joint and base_theta_joint are exactly 0.0000 on all 2,587 frames.
| Removed | episodes | frames | gripper travel |
|---|---|---|---|
| Gripper commanded 0.6000 open on every frame | 1 | 125 | 0.0000 |
| Gripper never opens past 0.3634 of the 0.6000 open command | 1 | 135 | 0.6183 |
| Removed | 2 | 260 | below 0.70 |
| Kept | 18 | 2,327 | 0.8199 to 0.8701 |
The removed media
Measured on the release. There is no video here, the two cameras are stored as one JPEG per frame at 6 fps, so each card is a single frame from the head or the gripper camera. The number on each card is the gripper command at that exact frame and the command travel of the take it comes from.
