Evaluation report, LeRobot folding
LeRobot publishes a curated folding corpus, high_quality_folding, alongside the full one. Measured against its own shipped progress score, the curated cut ends with the shirt unfolded 2.6 times more often than the untouched corpus it was drawn from. The reason is that it is not one dataset. It is three, joined end to end, and only the first was filtered.
The curated set is three datasets
Share of episodes ending below 0.90 progress, per block of 100 episode indices. A curated corpus should look flat. This one is flat and clean for a thousand episodes and then steps to 62%. Nothing about the task or the robot changes at that boundary.
The three populations, and what the cut is worth
Every rate carries a Wilson 95% interval, because a rate without its n is not a claim. Not folded means the progress score never got past 0.50 by the last frame. Not finished means it ended below 0.90.
| population | episodes | not folded | not finished | median length | present in full_folding |
|---|---|---|---|---|---|
| high_quality_folding, episodes 0 to 999 | 1,000 | 0.80% [0.41, 1.57] | 3.60% [2.61, 4.94] | 67 s | yes, all 1,000 |
| high_quality_folding, episodes 1000 to 1161 | 162 | 28.40% [22.01, 35.78] | 61.73% [54.05, 68.86] | 45 s | yes, all 162 |
| high_quality_folding, episodes 1162 to 1199 | 38 | 13.16% [5.75, 27.33] | 39.47% [25.60, 55.28] | 345 s | no, none of the 38 |
| high_quality_folding, all of it | 1,200 | 4.92% [3.83, 6.29] | 12.58% [10.83, 14.58] | 64 s | 1,162 of 1,200 |
| full_folding minus the cut | 4,526 | 1.24% [0.95, 1.60] | 8.17% [7.41, 9.01] | 72 s | by definition |
| full_folding, untouched | 5,688 | 1.92% [1.59, 2.31] | 8.81% [8.10, 9.57] | 70 s | by definition |
Read the first row against the last. On the first thousand episodes the curation is real and it works: 0.80% unfolded against 1.92% for the raw corpus, and the intervals do not overlap. Then read the fourth row against the last. Taken whole, the curated set is 2.6x worse than doing nothing at all. The filter did not fail. It was applied to a thousand episodes and two hundred more were appended after it.
Episodes 1162 to 1199 are not in full_folding
Matching every curated episode back to the corpus on content, that is length plus all sixteen per-joint spreads, 1,162 of 1,200 are found exactly and zero fingerprints collide. The 38 that are absent are a contiguous run at the very end, and their median length is 345 seconds against 70 for the corpus. They are a different recording regime, not a selection from this one.
Where attempts stopped
Progress score at the last frame, all 1,200 curated episodes, 20 bins. The vertical axis is cut so the tail stays visible, so the tallest bar is taller than it is drawn and both of the top two carry their counts. Most episodes finish cleanly. The tail is the whole argument, and it is thicker than the label on the box implies.
Some episodes are recording sessions, not episodes
The progress score across three episodes, each drawn from its own start to its own end rather than on a shared clock, with its real duration beside it. Episode 257 is what the task looks like when it works, one clean rise. Episodes 1162 and 1195 rise and collapse over and over. That is not the robot undoing its work. Each collapse is one take ending and the workspace being reset with a fresh garment, which the clips below show directly. So these are not episodes. They are sessions holding many takes end to end, recorded under a single episode index and a single instruction to fold a t-shirt.
| counting a reset as the score passing 0.80 and then falling below 0.20 | value |
|---|---|
| Curated episodes containing at least one reset | 37 of 1,200, 3.1% |
| Median length, multi-take against single-take | 354 s against 64 s |
| Episodes over 240 s that are multi-take | 29 of 46 |
| Most takes found inside one episode | 11 resets, episode 1162 |
This matters separately from the counting above, and it cuts the other way too. For a multi-take session the progress score at the last frame says where recording stopped, not whether the task succeeded, so it is the wrong measure for those 37. It is the right measure for the rest. Of the 59 curated episodes ending under 0.50, 4 are multi-take and 55 are single takes that simply did not finish, 52 of them under a minute long. The headline is not carried by the mis-segmented ones.
The clips
Cut from the base camera in the release itself, at the timestamps meta/episodes gives for each episode. The wrist cameras cannot show whether a shirt ended up folded, so they are not used here.
A take ending, next to a take working
Eight frames across each clip. Top is episode 1162 at the 276 second mark, where the score falls from 0.89 to 0.07. The arms carry the finished garment out of shot, the table is empty for a moment, and then two human hands arrive with the next shirt. Bottom is episode 257, a single take from start to finish. The score is behaving correctly in both. Only the episode boundaries are wrong.
What we did not find
A report that only finds faults is prosecuting, not measuring. Three things came back clean and are worth stating.
| check | result |
|---|---|
| Frozen joints, a channel held at one value for a whole episode | Clean. Only 3 of the 16 joints ever freeze, one episode each, out of 1,200. No dead channel, no padding slot, no dropped estimator. |
| Motionless frames, teleoperator dead time | Clean. Median 3.6% of frames per episode, and only 5 episodes are more than a fifth motionless. |
| Duplicate episodes inside full_folding | Clean. Zero colliding content fingerprints across all 5,688. |
What to do about it
Use episodes 0 to 999 and drop the rest. That removes 17% of the curated set and 76% of its unfinished folds, and leaves a corpus measurably cleaner than the raw one rather than worse than it. If the last 200 are wanted, they need the filter that the first 1,000 already had, not a rerun of the label.
Method, and what this is not
No model was trained for this page and no success rate is claimed. This is an audit of the data as shipped, nothing more. Every figure above comes from the trajectory files and from sarm_progress.parquet, which both datasets carry and neither dataset card mentions. That file holds a per-frame progress score from LeRobot's own reward model level2_sarm9. It is their signal, not ours, which is what makes it usable as evidence about their curation.
The score was sanity checked before being trusted. On a clean episode it rises once and monotonically, which is episode 257 in the chart above, and it is flat at 0.02 for an episode where the robot never moves. A score that behaved the same way on both would have been dropped.
Thresholds. A joint counts as frozen when its spread across a whole episode is at or below 1e-9, which is Gantry's own constant, well under anything a real joint does and well over float32 noise. A frame counts as motionless when no joint moved more than 0.1% of that joint's own full range, relative rather than absolute because the units are undeclared. Intervals are Wilson at 95%. Curated episodes were matched into the corpus on the pair of episode length and all sixteen per-joint spreads rounded to six decimals.
Not measured. No video was downloaded, so nothing here is a claim about what the cameras saw, only about what the arms did and what the shipped score says about it. 576 GB of video was left on the server. The subtask annotations were read and then set aside as unusable, since 749 of the 1,200 curated episodes carry none at all.