We trained the same enjoyment scorer twice, once on HRI CUES as released and once on an edited HRI CUES with the 112 exchanges the three expert annotators split hardest on removed, and the edited one scored 5.2 points higher.
A same-size random cut of 112 exchanges scored 17.0%, so the gain is from which exchanges were removed, not from removing exchanges.
The two ends of the scale gain most, and that is where the removed exchanges sit, because 56.4% of every 1 and 45.8% of every 5 in the release come from an exchange the cut takes out. The pooled lift clears zero, single classes at 120 rollouts mostly do not, and the chart says so.
Measured on the two released files, every one of the 590 exchanges carries three expert enjoyment scores, so every one has a spread, the highest of the three minus the lowest on a five point scale. Only 147 exchanges are unanimous, and on 112 of the 590, 19.0%, the panel splits by two points or more, as far apart as P25 turn 9 where the three scores are 5, 1 and 2. The cut takes those 112 and nothing else, so the 27 exchanges where all three experts scored 2 or lower stay in, because a turn everyone agrees went badly is signal and a turn nobody agrees on is not. It lands hardest at the two ends of the scale, taking 29.9% of the exchanges the panel put below 2.5 and 22.6% of the ones it put at 4.0 or above, against 8.7% of the 3.5 to 4.0 band.
| Removed, by who broke from the panel | exchanges | mean spread |
|---|---|---|
| all three experts wrote a different number | 72 | 2.14 |
| Annotator 1 two or more points from the other two | 29 | 2.00 |
| Annotator 3 two or more points from the other two | 7 | 2.00 |
| Annotator 2 two or more points from the other two | 4 | 2.00 |
| Removed | 112 | 2.09 |
| Kept | 478 | 0.69 |