Gantry

Evaluation report, TRLC DK1 t-shirt folding

We fine-tuned the same policy twice on Gongsta/trlc_tshirt_folding, the DK1 recording the TRLC Data Explorer lists first, once on all 532 takes as released and once with the 21 that hold still for the largest share of their frames removed, and the edited run scored 5.0 points higher over 720 rollouts.

Fine-tuned on full
25.3%
Folding success with all 532 takes.
CI 22.2 to 28.6 · n=720
Fine-tuned on edited
30.3%
Folding success with the 21 stillest takes removed.
CI 27.0 to 33.7 · n=720
Lift
+5.0 pp
From training on less data.
95% range +0.4 to +9.6
Removed
3.9%
21 of 532 takes, 18.5 minutes of recording.
33,337 frames of 1,106,053
Dead takes
4
Takes whose commanded gripper never opens or closes once.
the 4 shortest of 532 · 7 frames, 0.23 s

Success rate by task

fine-tuned on full TRLC DK1 t-shirt folding fine-tuned on edited TRLC DK1 t-shirt folding

A same-size random cut scored 25.4%, so the gain is from which takes were removed, not from removing takes.

Gain per task

krishkrosh gains most, and the cut took 12 of his 89 takes, more than the other three teleoperators lost between them. kesu, who lost 2 of 97, comes out flat. The pooled lift clears zero, no single operator group of 180 rollouts does on its own, and the chart says so.

What the edited version had removed

Measured on all 532 takes in the release. Each take carries one number, its held-still share, the fraction of its frames on which the twelve commanded arm joints move less than 0.01 radians in total from the frame before, and every take above twice the corpus median of 4.5% was dropped. 12 of the 21 removed takes come from one of the four teleoperators, krishkrosh, who recorded 89 of the 532, against 9 removed from the 443 the other three recorded. The 74 takes carrying the most gripper open and close commands all stay in, because a re-grasp is something a policy can learn from, and 53 of the 54 longest takes stay in, because slow is not wrong.

Removedtakesmedian held still
commanded gripper never opens or closes4100%
arms frozen for 3 s or more at the start811.9%
dead air spread through the take99.6%
Removed2110.2%
Kept5114.5%

The removed media

Measured on the release. Six takes the cut removes and two it keeps, cut from the top camera at the per-take offsets carried in the release metadata. The number on each card is that take's own held-still share, the fraction of its frames on which the twelve commanded arm joints do not move, in a release whose median take runs 68.3 s.

episode_000414removed
Fold the t-shirt, recorded by steveng. The whole take, start to end.
366 frames · 12.2 s · held still on 100% of frames · 0 gripper commands
episode_000023removed
Fold the t-shirt, recorded by edwardius. The right gripper command never changes value.
412 frames · 13.7 s · held still on 12.9% of frames · right gripper travel 0.000
episode_000364removed
Fold the t-shirt, recorded by krishkrosh. The clip starts where the take starts.
2,127 frames · 70.9 s · held still on 23.2% of frames · arms frozen for the first 13.1 s
episode_000395removed
Fold the t-shirt, recorded by krishkrosh. The clip starts where the take starts.
2,282 frames · 76.1 s · held still on 13.8% of frames · arms frozen for the first 7.5 s
episode_000452removed
Fold the t-shirt, recorded by edwardius. The clip starts where the take starts.
2,043 frames · 68.1 s · held still on 14.1% of frames · arms frozen for the first 5.2 s
episode_000376removed
Fold the t-shirt, recorded by krishkrosh. The clip starts where the take starts.
2,277 frames · 75.9 s · held still on 13.8% of frames · arms frozen for the first 4.5 s
episode_000404kept
Kept. The most gripper open and close commands of any take in the release, and every re-grasp stays in.
3,712 frames · 123.7 s · held still on 2.6% of frames · 56 gripper commands
episode_000228kept
Kept. An ordinary take by kesu, close to the median length of the release.
1,951 frames · 65.0 s · held still on 2.6% of frames · 26 gripper commands
TRLC DK1 t-shirt folding · 532 takes · 10 h 14 min · 4 operator groups × 180 rollouts × 2 arms Wilson 95% · cut = held still on 9.1% of frames or more