The same multimodal model fine-tuned twice, once on CFG-Bench as released and once on an edited CFG-Bench with the 3,281 multiple choice items that at least two of three text-only pickers already answer removed, then scored over 1,650 rollouts per arm, where the edited run reached 39.4% against 34.2% and won by 5.2 points.
A same-size random cut of 3,281 multiple choice items scored 34.5%, so the gain is from which items were removed, not from removing items. The eight small task types the cut nearly empties, 386 of their 450 multiple choice items gone, take almost all of the movement, and Factual Action Understanding is the one task that does not move at all.
The task types that gain most are the small ones the cut nearly emptied. The pooled lift clears zero, most single task types at 150 rollouts do not, and the chart says so.
Measured on the two annotation files, 15.3 MB, with no video bytes read. Every one of the 9,615 multiple choice items carries one number, how many of three text-only pickers land on its correct option, one scoring each option by word overlap with the release's own caption for that same video, one always taking the longest option, one always taking option A. Those three land on 47.2%, 38.9% and 25.9% of the set against a 20.0% floor, and 3,281 items are answered by at least two of them, 34.1% of the multiple choice set and 16.8% of the whole release, with 868 answered by all three. The cut falls hardest on the two tiers the release sells as higher-order reasoning, 121 of the 133 multiple choice Evaluative Judgment items and 164 of the 208 Intentional Understanding ones against 1,396 of 5,242 in Temporal-Causal Relation, while all 9,947 open-ended items are kept because they carry no options for the shape to leak through.
| Removed | items | videos |
|---|---|---|
| Caption match, longest option and position A all correct | 868 | 491 |
| Caption match and longest option | 1,152 | 777 |
| Caption match and position A | 900 | 592 |
| Longest option and position A | 361 | 272 |
| Removed | 3,281 | 1,211 |
| Kept | 16,281 | 1,351 |
Measured on the release. Each card is one video and the number is how many of the multiple choice items hanging off it the cut removes. The two kept cards are videos where nothing is removed, every item on them is answered by at most one of the three pickers, which is the shape a question is supposed to have.