379 demonstrations of ten tasks on a Panda. An audit of every episode against its own
instruction found 36 that do not do what they say. We fine-tuned
twice, once on all 379 and once on the 343 that do, same budget, same ten tasks, 100 rollouts
each. The edited version won.
Fine-tuned on full
62.9%
Task success with all 379 demonstrations.
CI 59.9 to 65.8 · n=1,000
Fine-tuned on edited
68.4%
Task success with the 36 flagged demonstrations removed.
CI 65.5 to 71.2 · n=1,000
Lift
+5.5 pp
From training on less data.
95% range +1.3 to +9.6
Removed
9.5%
36 episodes. 11 never complete the task, 25 complete part of it.
9,936 of 101,469 frames
Where they were
4 of 10
Tasks holding 32 of the 36 flagged episodes.
worst, soup and tomato sauce in basket 10 of 33
Success rate by task
fine-tuned on full LIBERO-10fine-tuned on edited LIBERO-10
A same-size random cut scored 63.4%, so the gain is from
which episodes were removed, not from removing episodes. The tasks that gained are the tasks
whose demonstrations were flagged.
Gain per task
Edited minus full, with its 95% range. The two tasks that clear zero are the two with
the most flagged demonstrations. Tasks with none flagged did not move, which is what should
happen when the edit only touches what is wrong.
What the edited version had removed
Measured, not a placeholder. Every one of the 379 demonstrations was watched at 1 fps by
gemini-2.5-flash and judged against the instruction it was recorded under. 36 were flagged.
They are not spread evenly. 32 of the 36 sit in four
tasks, and 10 of the 33 demonstrations of
soup and tomato sauce in basket are wrong, mostly because the operator swapped the plates or grabbed a red
mug that the instruction never mentions. Three tasks have no flagged demonstration at all.
Why the annotator flagged it
episodes
wrong object handled
22
clip ends before the task does
8
other
3
fixture never operated
2
right object, wrong place
1
Removed
36
Kept
343
The removed media
All 36 removed demonstrations, main camera, with the annotator's reason under
each. The last two are demonstrations the cut keeps, from the two most-flagged tasks, for
contrast.
episode 000 · mugs on left and right platesnot completed
The original instruction was to put the white mug on the left plate (robot-left plate) and the yellow and white mug on the right plate (robot-right plate). The robot instead placed the red mug on the robot-right plate and the yellow and white mug on the robot-left plate. The white mug was not manipulated.
214 frames at 10 fps · annotator, gemini-2.5-flash
episode 005 · mug on plate, pudding beside itpartial
The robot successfully placed the white mug on the plate. However, the robot did not move the chocolate pudding box to the right of the plate. Instead, it grasped a red mug after placing the white mug.
240 frames at 10 fps · annotator, gemini-2.5-flash
episode 009 · soup and cream cheese in basketnot completed
The robot was instructed to put the alphabet soup and the cream cheese box in the basket. The robot instead picked up an orange bottle and placed it in the basket, and then picked up the cream cheese box and began to place it in the basket. The alphabet soup was not manipulated.
302 frames at 10 fps · annotator, gemini-2.5-flash
episode 011 · mug on plate, pudding beside itpartial
The robot successfully placed the white mug on the plate. However, the chocolate pudding box was initially to the viewer's right of the plate (robot's left), and the robot moved it further to the viewer's right. If 'right of the plate' refers to the robot's right, the task was not completed as the box was moved to the robot's left (viewer's right). If 'right of the plate' refers to the viewer's right, the box was already in that general area, and the robot moved it to another position within that area, which is a weak interpretation of 'put to the right of'.
218 frames at 10 fps · annotator, gemini-2.5-flash
episode 013 · soup and tomato sauce in basketnot completed
The robot did not manipulate the instructed 'alphabet soup' or 'tomato sauce'. Instead, it picked up an 'orange juice carton' and a 'green and white can' and placed them into the basket.
285 frames at 10 fps · annotator, gemini-2.5-flash
episode 020 · both moka pots on the stovepartial
The robot successfully placed the first moka pot on the stove. However, the second moka pot was still being held by the gripper above the stove when the video ended, indicating it was not fully placed.
359 frames at 10 fps · annotator, gemini-2.5-flash
episode 021 · mug on plate, pudding beside itpartial
The first part of the instruction, 'put the white mug on the plate', was not completed. The robot picked up the red mug instead of the white mug, placed the red mug on the white mug, and then placed the white plate on the red mug, rather than placing the white mug on the plate. The second part of the instruction, 'put the chocolate pudding to the right of the plate', was completed.
264 frames at 10 fps · annotator, gemini-2.5-flash
episode 031 · bowl in bottom drawer, close itnot completed
The robot grasped the bowl and moved it towards the drawer, but the drawer was never opened, the bowl was not placed inside, and the drawer was not closed.
241 frames at 10 fps · annotator, gemini-2.5-flash
episode 041 · soup and cream cheese in basketpartial
The robot placed two cans into the basket. One of these cans might correspond to 'alphabet soup'. However, the 'cream cheese box' (assumed to be the small blue cube) was grasped but not placed into the basket by the end of the video. Additionally, an extra can was placed in the basket that was not explicitly requested. The exact identity of the cans as 'alphabet soup' is not visually confirmed.
236 frames at 10 fps · annotator, gemini-2.5-flash
episode 052 · mug on plate, pudding beside itpartial
The robot successfully placed the white mug on the plate. However, the video ends after the robot grasps and lifts the chocolate pudding box, without showing its final placement to the right of the plate.
237 frames at 10 fps · annotator, gemini-2.5-flash
episode 057 · both moka pots on the stovepartial
The task required placing both moka pots on the stove. Only one moka pot was successfully placed. The second moka pot was grasped and moved towards the stove, but the video ended before it was fully placed or released.
368 frames at 10 fps · annotator, gemini-2.5-flash
episode 071 · soup and tomato sauce in basketpartial
The robot successfully placed the alphabet soup in the basket. However, the robot did not pick up or place the tomato sauce in the basket. Additionally, the robot performed an extraneous action of picking up an orange box and placing it on the table.
312 frames at 10 fps · annotator, gemini-2.5-flash
episode 077 · soup and tomato sauce in basketpartial
The robot successfully placed the alphabet soup can into the basket. However, it failed to place the tomato sauce bottle into the basket. Additionally, the robot picked up and placed an orange juice carton, which was not part of the original instruction.
300 frames at 10 fps · annotator, gemini-2.5-flash
episode 082 · soup and cream cheese in basketpartial
The task required placing "both the alphabet soup" (implying two cans) and the cream cheese box into the basket. The robot placed the cream cheese box and only one of the two alphabet soup cans. Additionally, the robot placed a red bottle (tomato sauce) into the basket, which was not part of the instruction.
290 frames at 10 fps · annotator, gemini-2.5-flash
episode 085 · mugs on left and right platespartial
The original instruction asked to put the white mug on the left plate, but the robot placed a red patterned mug on the robot-left plate instead. The second part of the instruction, to put the yellow and white mug on the right plate, was completed correctly.
292 frames at 10 fps · annotator, gemini-2.5-flash
episode 088 · mugs on left and right platesnot completed
The original instruction was to 'put the white mug on the left plate and put the yellow and white mug on the right plate'. The robot did not manipulate the white mug at all. Instead, it placed a red mug on the robot-right plate and the yellow and white mug on the robot-left plate, which contradicts both parts of the instruction regarding the specific mug and target plate.
269 frames at 10 fps · annotator, gemini-2.5-flash
episode 101 · soup and tomato sauce in basketpartial
The robot picked up an orange juice carton instead of the tomato sauce bottle. One of the two specified objects (tomato sauce) was not manipulated.
278 frames at 10 fps · annotator, gemini-2.5-flash
episode 102 · soup and tomato sauce in basketpartial
The original instruction was to put both the alphabet soup and the tomato sauce in the basket. The robot placed the alphabet soup can in the basket, but it also placed an orange juice carton (a distractor) in the basket and did not place the tomato sauce bottle in the basket.
317 frames at 10 fps · annotator, gemini-2.5-flash
episode 107 · mugs on left and right platesnot completed
The task instructed to put the white mug on the left plate and the yellow and white mug on the right plate. Both mugs were already on their respective target plates (robot-left for the white mug, robot-right for the yellow and white mug). The robot moved the white mug from the robot-left plate to a robot-right plate and moved the yellow and white mug from its initial robot-right plate to another robot-right plate, thus not completing the instruction.
237 frames at 10 fps · annotator, gemini-2.5-flash
episode 121 · mugs on left and right platesnot completed
The robot first grasped a red-and-white mug, which was not specified in the instruction, and placed it on the robot-right plate. The instruction asked for the white mug to be placed on the left plate and the yellow and white mug on the right plate. The robot then grasped the yellow and white mug but did not complete its placement before the video ended.
292 frames at 10 fps · annotator, gemini-2.5-flash
episode 125 · mugs on left and right platesnot completed
The original instruction was to put the white mug on the left plate (robot-left plate) and the yellow and white mug on the right plate (robot-right plate). The robot placed the white mug on the robot-right plate and the yellow and white mug on the robot-left plate, effectively swapping the target plates for the two mugs.
242 frames at 10 fps · annotator, gemini-2.5-flash
episode 129 · mugs on left and right platesnot completed
The robot placed the white mug on the robot-right plate instead of the robot-left plate, and the yellow and white mug on the robot-left plate instead of the robot-right plate. The placements were swapped.
250 frames at 10 fps · annotator, gemini-2.5-flash
episode 140 · mug in microwave, close itnot completed
The robot placed the yellow and white mug inside the microwave while the microwave door was visibly closed, indicating a simulation error or impossible action. Furthermore, the robot opened the microwave door instead of closing it as instructed.
312 frames at 10 fps · annotator, gemini-2.5-flash
episode 160 · soup and cream cheese in basketpartial
The robot successfully placed the cream cheese box in the basket. However, instead of placing the 'alphabet soup' (which was likely one of the cans on the table), the robot picked up and placed an orange bottle into the basket.
258 frames at 10 fps · annotator, gemini-2.5-flash
episode 174 · soup and cream cheese in basketpartial
The robot placed the cream cheese box in the basket as instructed, but it placed a red bottle (hot sauce/ketchup) instead of the alphabet soup can into the basket.
244 frames at 10 fps · annotator, gemini-2.5-flash
episode 199 · soup and tomato sauce in basketpartial
The robot placed the alphabet soup can in the basket as instructed, but it also placed a distractor milk carton in the basket and did not place the tomato sauce bottle in the basket.
250 frames at 10 fps · annotator, gemini-2.5-flash
episode 211 · mugs on left and right platespartial
The first part of the instruction, 'put the white mug on the left plate', was completed. The second part, 'put the yellow and white mug on the right plate', was not completed as the robot only grasped the yellow and white mug before the video ended.
277 frames at 10 fps · annotator, gemini-2.5-flash
episode 212 · mug on plate, pudding beside itnot completed
The robot grasped the red mug instead of the white mug and placed it on the white mug, not on the plate. The robot also did not interact with the chocolate pudding box.
220 frames at 10 fps · annotator, gemini-2.5-flash
episode 222 · soup and cream cheese in basketpartial
The robot placed a red bottle into the basket instead of the instructed alphabet soup. The cream cheese box was correctly placed.
243 frames at 10 fps · annotator, gemini-2.5-flash
episode 238 · soup and tomato sauce in basketpartial
The original prompt requested to put both the alphabet soup and the tomato sauce in the basket. The robot successfully placed the tomato sauce, but instead of the alphabet soup, it picked up and placed an orange juice box into the basket.
340 frames at 10 fps · annotator, gemini-2.5-flash
episode 241 · soup and tomato sauce in basketpartial
The robot successfully placed the alphabet soup box in the basket. However, the video ends with the robot lowering the tomato sauce can into the basket, but the gripper is still closed, so the release of the tomato sauce can is not visibly completed.
274 frames at 10 fps · annotator, gemini-2.5-flash
episode 260 · soup and tomato sauce in basketpartial
The robot successfully placed the alphabet soup in the basket. However, it did not place the tomato sauce in the basket, as the video ended while the robot was still approaching it. Additionally, the robot placed an orange juice box in the basket, which was not part of the original instruction.
367 frames at 10 fps · annotator, gemini-2.5-flash
episode 309 · mugs on left and right platespartial
The original instruction was to "put the white mug on the left plate and put the yellow and white mug on the right plate". The robot placed a red mug on the robot-left plate instead of the white mug, which remained untouched. The yellow and white mug was correctly placed on the robot-right plate.
264 frames at 10 fps · annotator, gemini-2.5-flash
episode 325 · mug on plate, pudding beside itpartial
The robot successfully placed the white mug on the plate. However, for the second part of the instruction, "put the chocolate pudding to the right of the plate", the robot manipulated the red mug instead of the chocolate pudding box.
265 frames at 10 fps · annotator, gemini-2.5-flash
episode 355 · mug on plate, pudding beside itpartial
The first part of the instruction, 'put the white mug on the plate', was not completed by manipulating the white mug itself. Instead, the robot placed a red-and-white mug on the white mug, and then moved the plate underneath the white mug. The second part, 'put the chocolate pudding to the right of the plate', was successfully completed.
237 frames at 10 fps · annotator, gemini-2.5-flash
episode 359 · soup and tomato sauce in basketpartial
The robot placed the alphabet soup can in the basket as instructed, but it picked up an orange juice box instead of the tomato sauce bottle.
342 frames at 10 fps · annotator, gemini-2.5-flash
episode 018 · mugs on left and right plateskept
The robot places the white mug on the robot-left plate and the yellow and white mug on the robot-right plate.
276 frames at 10 fps · annotator, gemini-2.5-flash
episode 008 · soup and tomato sauce in basketkept
The robot successfully places both the alphabet soup can and the tomato sauce bottle into the basket.
259 frames at 10 fps · annotator, gemini-2.5-flash
lerobot/libero_10 · 379 episodes · 101,469 frames at 10 fps · Panda · annotations DaivdYuan/libero-10-execution-annotationsWilson 95% · cut = annotator-flagged episodes · preview, fine-tune numbers are placeholders