Evaluation report, lerobot/pusht

We fine-tuned the same policy twice on lerobot/pusht, once on the release as published and once with the 53 episodes that are too short to fill a single 100-step action chunk taken out, and the edited version scored 5.5 points higher across 640 graded rollouts.

Fine-tuned on full

28.0%

Stage success with all 206 episodes.

CI 24.6 to 31.6 · n=640

Fine-tuned on edited

33.4%

Stage success with the 53 short episodes taken out.

CI 29.9 to 37.2 · n=640

Lift

+5.5 pp

From training on 25.7% fewer episodes.

95% range +0.4 to +10.5

Removed

25.7%

53 of the 206 episodes are under 100 frames, so no frame in them can start a full action chunk.

4,319 of 25,650 frames · 7.2 of 42.8 minutes

Success flag

Frames out of 25,650 where next.success is true, while per-episode peak next.reward runs 0.813 to 0.949.

constant False on 206 of 206 episodes

Success rate by task

fine-tuned on full lerobot/pusht fine-tuned on edited lerobot/pusht

A same-size random episode cut scored 27.4%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

Rotate and reach gain most, and those are the two stages the removed episodes barely show, since the removed set travels a median 768 pusher units against 1,333 for the ones that stay. Holding the block on target to the last frame is flat, and the chart says so.

What the edited version had removed

Measured on all 206 episodes and all 25,650 frames of the release, read from the 104 KB episode metadata file and the 658 KB action parquet rather than a sample. Every episode gets one number, the share of its frames from which a 100-step action chunk runs past its last frame, and that share is 1.0 on 53 of the 206 episodes, so no frame in a quarter of this release can start a full chunk. Episode length runs 49 to 246 frames around a median of 122, a five-fold spread, and across the release 39.3% of the 2,565,000 action steps a 100-step chunk loader asks for fall past the end of their episode and have to be padded, 57.8% inside the 53 removed episodes against 35.5% in the 153 kept. The removed episodes are also the thinner demonstrations, a median 8.3 seconds against 13.7, a median 768 units of pusher travel against 1,333, and a median coverage gain of 0.838 against 0.878.

Removed	episodes	frames	median coverage gained
under 64 frames, short of a 64-step diffusion horizon as well	3	165	0.771
64 to 79 frames	19	1,347	0.826
80 to 99 frames	31	2,807	0.881
Removed	53	4,319	0.838
Kept	153	21,331	0.878

The removed media

Measured on the release. The six removed clips are cut from the packed 96 by 96 render at each episode's own from_timestamp and each one runs for exactly as long as its episode does, which is the point, every one of them is over before a 100-step chunk could finish. The shortest episode in the release is 49 frames, 4.9 seconds, against a median of 122 frames. The two kept clips are the release at its best, the longest episode and the highest peak coverage anywhere in it.

episode_000205removed

The shortest episode in the release, done in under half an action chunk.

49 frames · 4.9 s · peak coverage 0.825

episode_000182removed

Second shortest, and the reset already had the block 12% onto the target.

56 frames · 5.6 s · coverage gained 0.771

episode_000060removed

Six seconds of push, 559 units of pusher travel against a kept median of 1,333.

60 frames · 6.0 s · peak coverage 0.869

episode_000093removed

Reaches 0.895 coverage but stops 36 frames short of one chunk.

64 frames · 6.4 s · peak coverage 0.895

episode_000008removed

The reset placed the block 0.591 onto the target, so this demonstration only ever shows the last third of the push.

80 frames · 8.0 s · coverage gained 0.284

episode_000024removed

Starts at 0.547 coverage, the second least of the task shown anywhere in the release.

98 frames · 9.8 s · coverage gained 0.332

episode_000174kept

The longest episode in the release, a full approach, rotate and push from no overlap at all.

246 frames · 24.6 s · coverage gained 0.926

episode_000032kept

The highest peak coverage anywhere in the release at 0.949, and next.success is still false on every frame of it.

148 frames · 14.8 s · peak coverage 0.949

lerobot/pusht · 206 episodes · 25,650 frames · 4 stages × 160 rollouts × 2 arms Wilson 95% · cut = episodes under 100 frames, the default ACT chunk size