Gantry

Evaluation report, CMU Motion Capture Library, FBX

We fine-tuned the same text conditioned motion model twice, once on the CMU FBX set as released and once on an edited version with the 398 clips that name no motion or point at a missing file removed, and the edited one scored 4.9 points higher over 960 held out prompts.

Fine-tuned on full
31.9%
Prompt success with all 2,548 indexed clips.
CI 29.0 to 34.9 · n=960
Fine-tuned on edited
36.8%
Prompt success with the 2,150 clips left after the cut.
CI 33.8 to 39.9 · n=960
Lift
+4.9 pp
From training on less data.
95% range +0.6 to +9.1
Removed
15.6%
398 of 2,548 clips, 71.2 MB of 686.5 MB.
median label reach 201 · 2 for what stays
Labels kept
1,096
Of the 1,100 distinct labels in the release, still covered after the cut.
4 lost · 68 of 113 subjects untouched

Success rate by task

fine-tuned on full CMU Motion Capture Library, FBX fine-tuned on edited CMU Motion Capture Library, FBX

The eight prompts are label strings taken verbatim from metadata.csv, with the walking and housework families first because that is where the cut concentrated. A same-size random cut scored 32.4%, so the gain is from which clips were removed, not from removing clips.

Gain per task

The two prompts that gain most are stair and terrain walking, which is where the generic walk bucket sat. The pooled lift clears zero, most single prompts at 120 rollouts do not, and the chart says so.

What the edited version had removed

Measured on the release. Every one of the 2,548 rows in metadata.csv was scored by its label reach, the number of clips a text query for that exact label returns once case is normalised. The median clip has a reach of 3 and 702 clips carry a label no other clip shares, but 201 clips carry the literal string Unknown, 98 carry only the word clean and 94 only the word walk, so 393 clips sit in three buckets that name no motion at all. Dropping those, plus the 5 rows that point at FBX files the repo does not contain, removes 398 clips and 71.2 MB and still leaves 1,096 of the 1,100 distinct labels covered.

Removedclipslabel reach
label is the literal string Unknown, zero rows are blank so it is a written non-label201201
label is the single word clean, from two performers9898
label is the single word walk, while normal walk, walk slow, walk fast and bouncy walk are kept separate9494
metadata row points at an FBX file that is not in the repo537
Removed398201
Kept2,1502
CMU FBX · 2,548 metadata rows · 2,543 files · 8 prompts × 120 rollouts × 2 arms Wilson 95% · cut = label reach 50 or more, or the indexed FBX file is absent · release carries no frames