Gantry

Evaluation report, box-opener

SmolVLA fine-tuned twice, once on viamrobotics/box-opener as released and once on an edited version with the 19 fastest-paced episodes removed. Same budget, same rollout suite, 510 rollouts per arm. The edited version scored 5.7 points higher.

Fine-tuned on full
21.6%
Task success with all 110 episodes.
CI 18.2 to 25.3 · n=510
Fine-tuned on edited
27.3%
Task success with the 19 fastest-paced episodes removed.
CI 23.6 to 31.3 · n=510
Lift
+5.7 pp
From training on less data.
95% range +0.4 to +10.9
Removed
17%
19 of 110 episodes, every one of them labelled open box flaps.
3,782 frames · 6.3 of 49.7 minutes
Pace spread
2.5x
Fastest demonstration against slowest, same task string, same arm.
29.6 to 73.6 deg per second

Success rate by task

fine-tuned on full box-opener fine-tuned on edited box-opener

The release carries one task string, open box flaps, so the rollout suite grades that task against held-out episodes drawn from each third of the recording session. A same-size random cut scored 21.9%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

The session-end third is where the cut concentrated and it is where the gain lands. The pooled lift clears zero, the single conditions at 170 rollouts each mostly do not, and the chart says so.

What the edited version had removed

Measured on all 110 episodes of the release. Each episode gets one number, the commanded joint pace, which is the total absolute travel of the six joint targets across the episode divided by that episode's own duration. Every episode carries the same task string, open box flaps, yet pace runs from 29.6 to 73.6 degrees per second, and it climbs as the session goes on while episode length falls 27% from the first half of the session to the second, correlation -0.72 between episode index and length. The 19 episodes at 58 degrees per second or faster average 19.9 seconds against 28.6 seconds for the 91 kept, and 16 of those 19 sit in the last 36 episodes recorded.

Removedepisodesmean pace
rushed and clipped, under 18 seconds563.5 deg/s
rushed, 18 to 21 seconds863.9 deg/s
rushed at ordinary length, 21 seconds and over661.8 deg/s
Removed1963.1 deg/s
Kept9146.5 deg/s

The removed media

Measured on the release. Each clip is one episode of the scene camera, cut at that episode's own from_timestamp inside the packed video. The last number in every caption is that episode's commanded joint pace, against a corpus median of 48.8 degrees per second.

episode_104removed
The fastest take in the release. The arm is driven at two and a half times the pace of the slowest episode, for the same task string.
177 frames · 17.7 s · 73.6 deg per second
episode_103removed
Recorded immediately before 104 and almost as fast. Both sit in the last dozen episodes of the session.
190 frames · 19.0 s · 67.9 deg per second
episode_092removed
The ninety-third take of the session and already down to under 20 seconds, against a corpus median of 26.8.
198 frames · 19.8 s · 67.9 deg per second
episode_108removed
Second from last in the release. Its total joint travel is 1,033 degrees against a corpus median of 1,274.
164 frames · 16.4 s · 63.0 deg per second
episode_109removed
The final episode recorded, ten seconds shorter than the corpus median.
168 frames · 16.8 s · 62.8 deg per second
episode_093removed
The shortest episode in the release at 16.1 seconds. Its total joint travel is 960 degrees against a corpus median of 1,274.
161 frames · 16.1 s · 59.6 deg per second
episode_022kept
Kept. The longest take in the release and the largest joint travel of any episode, 2,457 degrees. Long is not the defect, rushed is.
621 frames · 62.1 s · 39.6 deg per second
episode_047kept
Kept. The slowest-paced episode in the release, which is what a deliberate demonstration of this task looks like.
372 frames · 37.2 s · 29.6 deg per second
viamrobotics/box-opener · 110 episodes · SmolVLA · 3 conditions × 170 rollouts × 2 arms Wilson 95% · cut = commanded joint pace at or above 58 deg per second