Evaluation report, Trossen AI LeRobot releases
We fine-tuned the same policy twice on the eleven Trossen AI LeRobot v2.1 releases, once on all 560 episodes as published and once with the 59 that keep recording after the last commanded motion or barely move the gripper on a task that asks for a grasp removed, and the edited run scored 25.7% against 20.5%, a gain of 5.2 points over 600 rollouts.
Success rate by task
A same-size random cut scored 20.9%, so the gain is from which episodes were removed, not from removing episodes.
Gain per task
Peg insertion and organize tools move most, and they are the two releases the cut emptied hardest, taking 31 of 51 and 23 of 83 episodes. Coffee cup, which the cut took nothing out of, does not move at all. The pooled lift clears zero, single tasks at 100 rollouts mostly do not, and the chart says so.
What the edited version had removed
Measured on the eleven Trossen AI LeRobot v2.1 releases, 560 episodes and 349,886 frames of teleoperation at 30 fps. The command quantum in the shipped action stream is 0.0003815 radians, so a frame in which no arm joint and no gripper is asked to move more than two counts of that is the robot being told to hold position, and 50,604 of the 349,886 frames, 14.5%, are held that way. Capture stops on a fixed timer rather than on the task, so the held frames pile up at the end, and 56 episodes carry five seconds or more of them after the last commanded motion. The cut is those 56 plus the 3 episodes whose gripper travel is under a quarter of its own task median on a task that asks for a grasp, 59 episodes dropped whole rather than trimmed because nothing in the release marks where the task finished.
| Removed | episodes | held at the end | gripper travel |
|---|---|---|---|
| timer runs on after the last commanded motion | 56 | 6.7 s | 99% |
| gripper never travels a quarter of its own task median | 3 | 0.0 s | 0.2% |
| Removed | 59 | 6.6 s | 99% |
| Kept | 501 | 0.0 s | 100% |
The removed media
Measured on the releases, one card per episode, from the overhead camera. The held figure is the run of frames at the end of the episode in which nothing on either arm is commanded to move more than two encoder counts, and the gripper figure is that episode's total gripper travel as a share of its own task median. The three handover cube cards come out of a repo whose meta/info.json does not parse as JSON, a missing comma after the trossen_subversion field, so the file has to be patched by hand before a standard LeRobot load will read it.
