Gantry

Evaluation report, GigaBrain-0.7 Sample

We fine-tuned the same policy twice on GigaBrain-0.7 Sample, once on the release as published and once with 234 episodes removed, and the edited run scored 37.0% against 31.8%, a gain of 5.2 points over 1,200 rollouts.

Fine-tuned on full
31.8%
Task success with all 1,372 episodes.
CI 29.3 to 34.5 · n=1,200
Fine-tuned on edited
37.0%
Task success with the worst 234 episodes removed.
CI 34.3 to 39.8 · n=1,200
Lift
+5.2 pp
From training on less data.
95% range +1.4 to +8.9
Removed
17%
234 of 1,372 episodes, 108,633 of 608,915 frames.
60 minutes of 5.6 hours
Constant everywhere
6 of 25
Action dimensions that never move across all 163,819 push button frames.
3 exactly flat · 3 under 1e-3 rad

Success rate by task

fine-tuned on full GigaBrain-0.7 Sample fine-tuned on edited GigaBrain-0.7 Sample

A same-size random cut scored 32.0%, so the gain is from which episodes were removed, not from removing episodes.

Gain per task

The blue button task gains most, and two of the woven basket tasks barely move. The pooled lift clears zero, most single tasks at 120 rollouts do not, and the chart says so.

What the edited version had removed

Measured on the release, every episode was scored on how much of its own action head is a constant, meaning a commanded dimension whose value spans under 1e-3 rad from the episode's first frame to its last. 6 of the 25 action dimensions never move across any of the 163,819 push button frames, three of them holding exactly one value, and 2 of 16 never move across all 445,096 pick and place frames, so a floor is baked into the release before any episode is scored. On top of that floor, 105 of the 360 push button episodes hold the entire right arm constant from first frame to last, and none hold the left arm constant, under one instruction per subset that never says which arm to use. The cut removes the 234 episodes that sit above their own subset's median, which is 4 or more constant dimensions of 16 for the agilex robot and 17 of 25 for the h01.

Removedepisodesaction head constant
push button, seven or eight right arm channels constant for the entire episode10417 of 25
pick and place, the left waist and the left gripper never move1204 of 16
pick and place, a waist joint and a gripper never move94 of 16
pick and place, six of the seven left arm channels never move18 of 16
Removed234108,633 frames
Kept1,138500,282 frames

The removed media

Measured on the release. Each clip is the window the episode occupies inside its packed mp4, read from the from_timestamp column of meta/episodes. The number in every caption is how many of that episode's own action dimensions hold a single value from its first frame to its last.

push_button_blue/episode_000006removed
Push the blue button next to the plate
586 frames · 17 of 25 dimensions constant · whole right arm still
push_button_red/episode_000004removed
Push the red button next to the plate
541 frames · 17 of 25 dimensions constant · whole right arm still
push_button_yellow/episode_000003removed
Push the yellow button next to the plate
684 frames · 17 of 25 dimensions constant · whole right arm still
pick_and_place/episode_000192removed
Put the orange into the yellow plastic mesh basket Subtask, raise the right hand.
323 frames · 8 of 16 dimensions constant · six of seven left arm channels still
pick_and_place/episode_000188removed
Put the orange into the yellow plastic mesh basket Subtask, raise both hands.
632 frames · 4 of 16 dimensions constant · left waist and left gripper still
pick_and_place/episode_000611removed
Put the apple into the woven basket Subtask, raise both hands.
612 frames · 4 of 16 dimensions constant · right waist and right gripper still
pick_and_place/episode_000816kept
Put the lemon into the woven basket Subtask, raise both hands.
654 frames · 2 of 16 dimensions constant · both grippers actuate
pick_and_place/episode_000388kept
Put the green pepper into the woven basket Subtask, raise both hands.
621 frames · 2 of 16 dimensions constant · both grippers actuate
GigaBrain-0.7 Sample · 1,372 episodes · GR00T N1.6 · 10 tasks × 120 rollouts × 2 runs Wilson 95% · cut = above its own subset's median constant share