Gantry

Evaluation report, Ego-Exo Manufacturing

Third Origin's synced release. 40 takes of shoe-factory work, 36.2 hours of head-mounted video, each take narrated action by action and rated by an expert. We used it as the ego pre-training stage for an SO-ARM101 policy twice, once as released and once with 143 minutes removed, then ran both on six bench tasks, 127 rollouts each. The edited version won.

Pre-trained on full
23.0%
Bench success after the ego stage on all 36.2 hours.
CI 20.1 to 26.1 · n=762
Pre-trained on edited
27.4%
Same stage, 143 minutes removed.
CI 24.4 to 30.7 · n=762
Lift
+4.5 pp
From training on less video.
95% range +0.1 to +8.8
Removed
6.6%
143 of 2175 minutes. 58 of them are a camera that had been taken off.
52 segments · 3 whole takes
Expert rated
37 of 40
Takes the annotator calls expert or proficient. Three are beginners.
median rating 5.0 of 5

Bench success by task

ego stage on full corpus ego stage on edited corpus

A same-size random cut scored 23.6%, so the gain is from which minutes were removed, not from removing minutes. Fold and lid, the two tasks with the most matching footage, gained most.

Gain per task

Edited minus full, with its 95% range. The pooled lift clears zero. Single tasks at 127 rollouts mostly do not, and the chart says so.

What the corpus is

Measured from the release. 40 synced ego and exo takes, 23,650 narrated actions with a timestamp, an expert rating on speed, precision and fluency for every take. 15 takes are rated expert, 22 proficient, 3 beginner. Scenarios are shoe assembly 21, sole attachment 9, shoe packaging 8, stitching 1, shoe finishing 1. The seller already grades its own workers, which is rare, and it makes the corpus a clean test of whether our score agrees with a human expert.

What the edited version had removed

Measured, from the annotations. The narrator marks moments where the wearer is not doing the task, and in one take writes that the camera was removed at 23 minutes while the file ran on for 57 more. Those minutes plus the moments spent waiting, fetching, being taught or talking, plus the three takes rated beginner, are the cut. The three takes with a minor annotated mistake are kept; one smear of glue in 107 minutes is not a reason to lose the take.

Why it was removedsegmentsminutes
camera off, still recording357.7
waiting, fetching, being taught, talking4914.8
whole take, annotator rates the worker a beginner370.3
Removed142.8
Kept2032

The removed media

Head camera, twelve seconds each, at the moment the annotation names. Six removed, two kept for contrast.

group06 · 25:00removed
The worker took the camera off at 23:38 and the take kept recording for another 57 minutes. This is 25 minutes into that.
Folding and attaching shoe box cover flaps · rated 5.0 · camera off, still recording
group01 · 00:02removed
An older worker is teaching the camera wearer the fold. The annotator notes it took longer for that reason.
Shoe box packaging paper folding and placemen · rated 4.3 · being taught
group01 · 07:25removed
Batch finished. The annotator notes the wearer then fetched more paper from elsewhere, and a coworker steps in. No manipulation in the interval.
Shoe box packaging paper folding and placemen · rated 4.3 · fetching materials
group64 · 04:10removed
Rests both hands on the table. The annotator's note reads, waiting for parts to arrive on the conveyor.
Binding shoe upper to sole with securing stra · rated 4.33 · waiting for parts
group07 · 03:47removed
Beginner take, rated 3.3 of 5. Here the worker is talking and nothing happens for the interval.
Folding protective paper for shoe box packagi · rated 3.33 · talking, no action
group41 · 01:00removed
The lowest-rated take in the corpus, 3 of 5 on speed, precision and fluency. Attaching mold strips.
Attaching shoe mold strips around the shoe ed · rated 3.0 · beginner take, rated 3.0
group20 · 10:00kept
Grinding a sole on the buffing machine. Rated 5 of 5 on every dimension, 1,903 narrated actions in 99 minutes.
Grinding shoe sole on buffing machine · rated 5.0 · expert, 5.0
group19 · 10:00kept
Hammering midsole to outsole. Rated 5 of 5. The kind of take the corpus is mostly made of.
Sole pressing - hammering midsole and outsole · rated 5.0 · expert, 5.0
third-origin/ego-exo-manufacturing-synced · 40 takes · 36.2 h ego · 1080p 30 fps · Apache 2.0 Wilson 95% · 6 tasks × 127 rollouts × 2 arms