Gantry
Gantry · dataset evaluation · 20 September 2026

214 of the 550 published episodes carry an empty instruction string

allday-technology/Locate-pickup-measuring-cup-2, measured from the public release on 20 September 2026.

the instruction field is empty

What is in the release

Every row is read out of the release. Nothing here is estimated and nothing is rounded up.

public releases read9
episodes across them550
releases whose only task string is the empty string3
episodes in those three214, which is 38.9%
downloads on those three462
Locate-pickup-measuring-cup-2100 episodes, tasks.jsonl holds one row, task is ""
Locate-pickup-measuring-cup63 episodes, same single empty row
Locate-pickup-soda-can51 episodes, same single empty row
releases that do carry a written instruction6
of those six, releases that write one task two ways3
locate_sauce_bottle_mergedlocate sauce bottle on 38 episodes, locate the sauce bottle on 35
pick-up-color-measuring-cuppick up the measuring cup with colored handle on 31, Pick up measuring cup on 30
cupHolderTrainingDatathree strings for one behaviour on 21 episodes, 1 and 10 and 10
releases where a text conditioned policy reads the instruction cleanly3 of 9

Why this matters

The instruction is the only field a language conditioned policy reads, and on 214 episodes it is the empty string. Those 214 do not become unlabelled, they become one label. Pooled with anything else they read as a single task taught 214 times, and the three behaviours in them, measuring cup, second measuring cup and soda can, collapse into each other. The six releases that do carry instructions show this is an export gap rather than a house convention, and three of those six spell one task two or three ways, so of nine releases three read cleanly. The fix is a string per release, not a re-collection.

How to check this without taking our word for it

curl the file meta/tasks.jsonl in Locate-pickup-soda-can. It is one line and the task field is "". Do it again in pickup-sauce-bottle-smolvla and the task field is pick up the sauce bottle. Then count distinct values of tasks in meta/episodes.jsonl for locate_sauce_bottle_merged and get two, at 38 and 35.

What a full grading adds

This page is what the release tells you on its own. A full grading removes the part that scores worst, retrains on what is left, and reports whether the smaller corpus scores higher, with rollout counts and intervals. The one result measured on physical hardware is Stera-10M, two arms on a four arm SO-ARM101 bench, 21.9% against 36.9% task success across 1,270 rollouts. The Stera-10M report.