Gantry

Evaluation report, SABER-10K

GR00T N1.6 fine-tuned twice, once on SABER-10K as released and once on an edited version with the lowest-scoring quarter removed. Same budget, same ten retail tasks, 100 rollouts each. The edited version won.

Fine-tuned on full
29.3%
Task success with all 10,000 episodes.
CI 26.6 to 32.2 · n=1,000
Fine-tuned on edited
35.2%
Task success with the worst 2,500 episodes removed.
CI 32.3 to 38.2 · n=1,000
Lift
+5.9 pp
From training on less data.
95% range +1.8 to +10.0
Removed
25%
2,500 episodes, 2,140 of them from the egocentric stream.
684,000 frames
Tasks up
9 of 10
The tenth was already at 100% on both.
largest, board duff +9

Success rate by task

fine-tuned on full SABER-10K fine-tuned on edited SABER-10K

A same-size random cut scored 30.1%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

The tasks that gained most are the ones the full corpus had left flat. The pooled lift clears zero; most single tasks at 100 rollouts do not, and the chart says so.

What the edited version had removed

Every episode was scored by how well its own frames predict its own actions, against a shuffled copy of itself. The lowest quarter was dropped. It is not a random quarter. 2,140 of the 2,500 removed episodes come from the egocentric stream, and in 1,900 of those the latent action is near constant for more than 80% of the clip while the camera keeps moving. They are aisle-walking segments labelled as manipulation. There is no action in them for a policy to learn, and the score found them without being told.

Removed quarterepisodesmargin
aisle walking, near-constant latent action1,900−4%
hand out of frame for most of the clip312−1%
retarget failure, joints pinned at limits204−6%
other low-margin84+2%
Removed2,500−3%
Kept7,500+14%

The removed media

Measured on the release. The egocentric stream's action is a four-token code, each token 0 to 7. 565 of its 5,000 episodes hold the same code on 80% or more of their frames, and the median episode repeats it on 71%. Seven in ten of its task strings describe walking, turning or pushing a cart. These six are the most constant in the whole stream. The last clip is one the cut keeps, for contrast.

stream1/episode_001120removed
Push the empty cart slowly past fruit displays.
240 frames · action unchanged on 97.5% of them · 3 distinct codes
stream1/episode_000611removed
Continue to wait in the queue.
300 frames · action unchanged on 93.3% of them · 8 distinct codes
stream1/episode_000348removed
Walk down the produce aisle, holding the basket and shopping list. They navigate past a small child who is also walking in the aisle.
300 frames · action unchanged on 92.6% of them · 4 distinct codes
stream1/episode_002727removed
Hold their phone with the payment application open, preparing to scan the QR code for payment.
150 frames · action unchanged on 92.6% of them · 7 distinct codes
stream1/episode_000853removed
Side-steps slowly in the aisle, holding a basket and scanning the shelves.
300 frames · action unchanged on 92.3% of them · 6 distinct codes
stream1/episode_001207removed
Walk forward in the produce section, passing a display of watermelons and other fruits. A customer walk past them.
300 frames · action unchanged on 92.6% of them · 7 distinct codes
stream1/episode_004427kept
Hold the white can, inspecting it briefly, and then place it back on the shelf.
270 frames · action unchanged on 3.7% of them · 63 distinct codes
SABER-10K · 10,000 episodes · GR00T N1.6 · 10 tasks × 100 rollouts × 2 arms Wilson 95% · cut = lowest-margin 25%