Evaluation report, CaliBench
We fine-tuned the same video model on CaliBench as released and on an edited CaliBench with the sixteen scene and model cells that fewer than half read out removed, and the edited one scored 5.0 points higher over 1,350 generations per arm, ahead on 7 of 9 scenes.
Success rate by task
A same-size random cut of 512 rollouts scored 29.8%, so the gain is from which cells were removed, not from removing rollouts. Walking and Dice are two of the three scenes the cut never touches, 3.1% and 1.6% of their generations are illegible, and they are the two scenes that do not gain.
Gain per task
The scenes that gain most are the ones the cut emptied. The pooled lift clears zero, most single scenes at 150 generations do not, and the chart says so.
What the edited version had removed
Measured on the committed outcome records for all 54 scene and model cells, 1,728 rollouts at 32 per cell. Each cell was scored by one number, how many of its 32 generations carry an outcome the extractor could read, and the cut takes every cell under half. That line sits in the widest gap in the whole distribution, the best removed cell reads out 12 of 32 and the worst kept cell reads out 18, so nothing sits near it. The rule drops 29.6% of the corpus and takes 447 of the release's 611 illegible generations with it, while 1,052 of the 1,117 legible outcomes stay, and the benchmark's own human labels agree, an annotator could read an outcome in 8 of the 46 sampled rollouts inside the cut against 99 of 116 outside it.
| Removed | rollouts | legible |
|---|---|---|
| Cosmos3-Super, six of its nine scenes | 192 | 30 |
| Veo 3.1, pendulum, cards and the physical board | 96 | 13 |
| Runway Gen-4.5, pendulum, cards and roulette | 96 | 8 |
| HappyHorse-1.0, pendulum and cards | 64 | 12 |
| WAN-2.7 and SeeDance-2.0, pendulum only | 64 | 2 |
| Removed | 512 | 65 |
| Kept | 1,216 | 1,052 |
The removed cells
Measured on the release. Each card is one scene and model cell shown on the single conditioning frame all 32 of its generations started from, and the number is how many of those 32 produced an outcome the extractor could read. Three cells produced nothing readable at all and every one of them is the double pendulum. The two kept cards are the opposite failure, fully readable and collapsed onto one outcome, which is a mistake a model can be trained against rather than an absence.
