Gantry

Evaluation report, Community Curated v0

SmolVLA fine-tuned twice, once on Community Curated v0 as released and once on an edited version with the 1,365 episodes whose cameras show the hand and the object together on under half the labelled frames removed. Same budget, same eight task strings taken from the release itself. The edited version scored 5.1 points higher over 960 graded rollouts.

Fine-tuned on full
32.4%
Task success with all 52,507 episodes.
CI 29.5 to 35.4 · n=960
Fine-tuned on edited
37.5%
Task success with the 1,365 low-visibility episodes removed.
CI 34.5 to 40.6 · n=960
Lift
+5.1 pp
From training on less data.
95% range +0.8 to +9.3
Removed
2.6%
1,365 of 52,507 episodes, spread over 139 of the 981 sub-datasets.
628,515 frames · 5.8 hours
One camera
12.6%
Share of the one-camera episodes the cut takes, against 1.8% at two cameras.
5,584 one-camera episodes · 701 removed

Success rate by task

fine-tuned on full Community Curated v0 fine-tuned on edited Community Curated v0

A same-size random cut scored 32.6%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

The chess move loses 449 episodes to the cut, more than any other task string in the release. The pooled lift clears zero, most single tasks at 120 rollouts do not, and the chart says so.

What the edited version had removed

Measured on the whole release, 981 sub-datasets and 52,507 episodes and 229.5 hours, summed from every meta/info.json. Each of the 50,592 episodes carrying a canonical judgment gets one number, the share of its judge-labelled frames in which the gripper and the manipulated object are both in shot, and 1,365 episodes fall below half, 160 of them at zero, where the work is never in shot at all. This is not a re-run of the release's own keep and discard field, because 628 of those 1,365 carry a keep verdict. It lands disproportionately on one-camera recordings, which are 11.0% of the judged corpus and 51.4% of the removed part.

Removedepisodesmean on-camera
object and gripper never both in shot1600%
in shot on under a quarter of the labelled frames12716%
in shot on a quarter to half of them1,07837%
Removed1,36530%
Kept49,22795%

The removed media

Measured on the release. The number on each card is that episode's on-camera share, the fraction of its judge-labelled frames in which the gripper and the object are both in shot, and every removed card below sits at zero. The last two cards are kept on purpose, including one the release's own judge marked review and logged five events against, because an imperfect episode the camera saw is something a policy can learn from and an empty frame is not.

sid003/something2 ep 4removed
Pick and place the block
1 camera · 508 frames · 11 labelled frames · 0% on-camera · all 4 judged episodes here are cut
dopaul/1500_chess_moves ep 1083removed
Move the chess piece from red to blue
1 camera · 373 frames · 8 labelled frames · 0% on-camera · 449 of this sub-dataset's 1,496 judged episodes are cut
Gano007/so100_lolo ep 8removed
Grasp a box and put it in the bin.
2 cameras · 897 frames · 20 labelled frames · 0% on-camera · gripper in shot on neither camera
JipJ/dice_project ep 17removed
Grab the blue dice
1 camera · 376 frames · 8 labelled frames · 0% on-camera · 14 of this sub-dataset's 40 judged episodes are cut
ganker5/so100_action_20250403 ep 11removed
Grasp a lego block and put it in the bin.
2 cameras · 423 frames · 9 labelled frames · 0% on-camera · all 16 judged episodes here are cut
T1g3rGE/so100_pickplace_small_20250407_171912 ep 6removed
Grasp items from right and place in black tray on the left
1 camera · 1,793 frames · 20 labelled frames · 0% on-camera · 59.8 s, the longest episode in the removed part
dopaul/1500_chess_moves ep 1359kept
Move the chess piece from red to blue
kept · 1 camera · 372 frames · 8 labelled frames · 100% on-camera · same task string as the second card
Gano007/so100_lolo ep 1kept
Grasp a box and put it in the bin.
kept · 2 cameras · 897 frames · 20 labelled frames · 100% on-camera · 5 logged events, verdict review
Community Curated v0 · 52,507 episodes · SmolVLA · 8 tasks × 120 rollouts × 2 arms Wilson 95% · cut = on-camera share below 0.50