Gantry
Gantry · dataset evaluation · 16 September 2026

29,839 of the 40,548 episodes in Open-H-Embodiment carry a uuid where the instruction should be

nvidia/PhysicalAI-Robotics-Open-H-Embodiment, measured from the public release on 16 September 2026.

the language instruction is a chunk identifier

What is in the release

Every row is read out of the release. Nothing here is estimated and nothing is rounded up.

LeRobot roots in the repo23
episodes across the release40,548
episodes whose task field is a uuid and a frame offset29,839
share of the release73.6%
roots affected4 of 23, all under Surgical/cmr_surgical
roots carrying real instructions19
distinct task strings inside the affected roots29,508
distinct task strings per episode there0.99
an actual task string49c39290-911e-4aec-8178-ee2409ff7854_3600
a task string from a root that is fineTie a surgical knot in the suture.
source recordings behind the prostatectomy root423
chunks cut from each recordingmedian 18, longest 122
modal episode length in the affected roots3,600 frames
declared frame rate there60, so exactly 60 seconds a chunk
offsets that are an exact multiple of 3,600, prostatectomy80.4%

The same data provider's other two roots, dry_box and peg_transfer, carry ordinary labels such as cutting. So this is not a house style and not a convention of the corpus. It is four roots out of twenty three.

Why this matters

In a LeRobot release the task field is the natural language instruction, and it is the string a language conditioned policy reads. For 73.6% of this release that string is a unique identifier that describes nothing and identifies the episode exactly. A model cannot generalise from it, and because it is unique per episode it is available as a memorisation shortcut instead. The loss will look healthy while that happens.

How to check this without taking our word for it

Open meta/episodes.jsonl under any of the four Surgical/cmr_surgical roots and read the tasks field, then open the same file under Surgical/hamlyn/knot_tying and compare. No credentials, no download of the parquet.

What a full grading adds

This page is what the release tells you on its own. A full grading removes the part that scores worst, retrains on what is left, and reports whether the smaller corpus scores higher, with rollout counts and intervals. The one result measured on physical hardware is Stera-10M, two arms on a four arm SO-ARM101 bench, 21.9% against 36.9% task success across 1,270 rollouts. The Stera-10M report.