453 episodes, 914,320 frames, 8.5 hours of bimanual SO-101 teleoperation, read frame by frame. No model was trained and nothing here is estimated. Every number below is an arithmetic statement about the release as shipped, and the ones that matter were recomputed by a second pass whose job was to break them.
This is not a quality opinion. It is a bug, it is silent, and it has probably been in every training run so far.
The file data/chunk-000/file-038.parquet contains a 5,380-frame
block labelled episode_index 236. The real episode 236 is a different 2,390-frame
recording in file-039, and that is the one meta/episodes describes.
The extra block is an orphan: a genuine 179-second recording that no metadata row points at.
It is not a copy of anything, and we checked it against all 453 other episodes.
Because the loader addresses frames by position, those 5,380 unlisted rows push every later episode's window out of alignment. We ran lerobot 0.6.1 against the directory to see what actually comes back rather than reasoning about it:
episode_index,
frame_index, index range and length all match the metadata exactly, while
the frames themselves belong to the orphan and differ from the real episode by up to 189 units.
No assertion or shape check can catch this.episodes=[236] returns 7,770 rows for a declared
length of 2,390, stitching two unrelated takes end to end.The effect is deterministic, not random, which is worse: it reproduces identically on every run and so looks like stable behaviour. For imitation learning with action chunking, a chunk that straddles a boundary asks the model to continue one recording using the next one's actions. For the progress model described below, every curve would be computed over the wrong span.
The fix is one line of bookkeeping. Delete the 5,380-row block from
file-038.parquet, or give it its own episode index and add a metadata row. Then
sum(meta.length) and the row count both come to 914,320 and everything downstream
realigns. Until then, no result measured on this dataset means what it appears to mean.
Stillness before the first commanded motion, per episode. 96.7% of episodes have more than a second of it; the mean is 6.3 s and the longest is 60 s. During these frames the action vector is bit-identical row to row, so this is not slow motion being missed by a threshold. We swept the threshold from 0 to 5 and the result is unchanged below 0.1.
Stillness after the last commanded motion. 84% of episodes end with over a second of it, mean 2.9 s. Together the two ends are 13.6% of the corpus, 69 minutes.
Stated as a hypothesis, because we did not run the experiment that would settle it.
Your policy folds once and stops. Nearly every episode in its training data ends with the arm commanded to hold position for two to three seconds after the last real motion. A behaviour-cloning objective has no way to distinguish "hold because the task is finished" from "hold because that is what the data does after a fold", and the hold is the last thing the model sees before every episode boundary.
We can say the defect is there and that it is pervasive. We cannot say from this data alone that it causes your plateau, and we would rather tell you that than sell you a story. It is cheap to test: retrain on the trimmed corpus and see whether the arm continues past the first fold. That is the one experiment we would run first.
One thing it is not: the corpus does not teach single-fold behaviour through the gripper. Median 14 grasp-release cycles per episode, 73.5% of episodes with 10 or more. The demonstrations contain plenty of folds. Whatever stops the policy, it is not that it never saw a second grasp.
Cut from the release at the offsets its own metadata gives, so each clip can be checked against the episode index rather than taken on trust. Red is removed, green is kept.
Each rule is a measured property of the episode's own frames. No rule uses a model, and none uses the task string except the off-task one.
| Rule | Episodes |
|---|---|
| Dropped, any rule | 40 |
Twelve of those are episodes 0 to 11, whose task string reads "Empty the desk by picking up each item and putting it in the box." That is table clearing, in a repository named folding_laundry.
Kept episodes grouped by the garment actually named. A language-conditioned policy sees "Fold the cloth on the table" and "Folding the cloth on the table" as two different instructions for one behaviour.
Measured against what that method needs, since the annotation spend depends on it.
| What ARM needs | What the release has |
|---|---|
| A completion signal to anchor the progress curve | Nothing. Every column in the release checked against success, reward, outcome, quality, label and twelve other patterns. Zero hits. Task strings state the goal, never the result. Someone has to define completion before a curve can be accumulated backwards from it. |
| A stagnant class worth learning | 16.9% of frames are mechanically stagnant, but 88% of that is recording dead air at the two ends. Genuine mid-task stagnation is 2.0% of frames, 10 minutes, in 82 of 453 episodes. Trim the ends first or the class is mostly an artefact of when record was pressed. |
| The head to learn from vision, not position | A shortcut exists. Position within the episode alone predicts the stagnant label at AUC 0.88. Untrimmed, a per-frame head can score well by learning "near an end means stagnant" without looking at the image. |
| A regressing class | Arm-pose self-revisits are rare: 372 of 453 episodes contain none at a 5-unit radius. But this measures joint positions only, and it is blind to the regression that matters here, cloth slipping from the gripper or a fold coming undone while the arm keeps moving. That is only visible in video. Do not conclude the class is empty; conclude it cannot be found this way. |
| Per-frame alignment | A uniform 4-frame, 133 ms lag between command and measured state, consistent across all 12 joints and all episodes. Shift the head by it, or the labels sit 4 frames ahead of the pixels that justify them. |
| Episode length as a difficulty proxy | Not usable. Within one task string the spread reaches 9.4x, and the same instruction maps to a 4x range in fold count, so the language label cannot teach a stopping criterion either. |
The practical read: a third to a half of the annotation pass is mechanical and should not be paid for by the hour. Trimming the ends and auto-labelling provable stagnation leaves the annotator the part that genuinely needs eyes, which is progressing against regressing on the cloth.
| Finding | Scope | Note |
|---|---|---|
| Left arm freezes bit-identical while the right works | 20 eps | Longest 59.7 s, ep 312. Zero equivalent cases on the right arm, p = 3.1e-5. Worth asking whether one leader arm drops out. |
| Episodes with no commanded motion at all | 6 | 94, 99, 232, 337, 353, 406. Episode 232 is 179 s of it. In four of the six the robot is still moving while the command is frozen, which is a stale command stream, not an empty recording. |
| Left gripper commanded over a 40% smaller range than the right | all | Median per-episode range 30.6 against 52.9, stable across the whole corpus. A calibration difference, not a fault, but it is asymmetric training signal. |
| Episodes cut off at a ~180 s recording ceiling | 4 | They end mid-motion. The fold is not finished in the data. |
Global index column is not unique | 10,760 rows | Indices 458534 to 463913 each appear twice. Same root cause as the loader bug. |
| Episode-start snap transient | 15 eps | First ~8 frames show the follower jumping from where the previous episode left it. Trim the head and it disappears. |
| No duplicate or near-duplicate episodes | 0 | Checked exhaustively. This axis is clean. |
| Tracking is clean | — | Minimum action-to-state correlation 0.97 at the 4-frame lag. Do not spend engineering time on servo tuning; it is not the problem. |
Ordered by what the measurement says is thin, not by what is easy to film.
So any line above can be checked or disputed.
Every data file was read: 73 parquets, 919,700 rows, of which 914,320 are the canonical 453 episodes and 5,380 are the orphan block. Per-frame motion is the sum of absolute change across the 12 action dimensions between consecutive frames. Stillness thresholds were swept from 0 to 5 units and the dead-air result is bit-identical below 0.1, because those frames are exactly repeated rather than slowly moving. The loader behaviour was established by running lerobot 0.6.1 against the directory, not by reading its source.
The loader result was then reproduced a second time, from freshly written code, against lerobot 0.6.1 in a clean environment. It returns 919,700 rows, the window metadata assigns to episode 236 comes back bit-identical to the orphan block (maximum absolute difference 0.0) and differs from the real episode by 189.15, 216 later windows span the wrong episodes, and 5,380 rows are never addressed. Every figure in this report was re-derived the same way before publication.
Each headline was also handed to an independent pass instructed to refute it from the raw files. That pass confirmed the dead-air, orphan, dead-episode and asymmetry findings, and corrected three of ours: our frame denominator double-counted the orphan, our grasp-cycle counter over-counted by about 25x and has been dropped, and our first grouping of the task strings was too coarse. Claims that survived only under one threshold are marked as such above rather than presented as settled.