2026.09.09 2026.09.09

Teleoperation Data for Imitation Learning: Recording 50 Episodes That Work

swiftwand

The arms are built and connected. Nothing has been taught yet. What comes next is the part nobody photographs: you move the leader arm through the same task fifty times while the software writes it down.

This is the stage where projects quietly fail. Not because the model is wrong, but because the recordings were inconsistent in a way that only becomes visible after training. This article organises what the official recording procedure actually says, and then reads what the numbers in it mean. This site has not recorded on hardware, so what follows is documentation organised for use, not a session report. Checked 2 September 2026.

Calibration comes first, and it is not optional
Teleoperation, and one environment caveat
Cameras: what the model cannot see, it cannot learn
How many, and how varied
Boring is the quality metric here
The counting trap when you resume
Replay before you train
Where the data sits in the whole pipeline
What to take away
Sources

忍者AdMax

Calibration comes first, and it is not optional

Before anything is recorded, both arms are calibrated. The reason is easy to state. The model outputs joint angles — but the same thirty degrees corresponds to a different physical pose on each machine, depending on assembly tolerances and where the servo horn ended up. Calibration absorbs that per-machine offset and puts everything on a shared coordinate frame.

Skip it and you get a dataset that describes a robot slightly different from the one you own.

Teleoperation, and one environment caveat

With calibration done, teleoperation is a single command pairing the follower and the leader by type and port. Moving the leader moves the follower, and the loop you will spend the next few hours in is established.

There is also a keyboard-driven mode, and it is fussier about your environment. The documentation states that keyboard teleoperation needs a global key backend, so it works only on an X11 session, a Windows desktop, or macOS with Accessibility and Input Monitoring granted — not on Wayland or headless sessions. The control-flow keys for starting and re-taking a recording are different: those work on X11, Wayland and headless or SSH sessions alike.

Cameras: what the model cannot see, it cannot learn

A vision-conditioned model needs camera frames in the recording. The official configuration examples show a single camera facing the workspace, and a two-camera setup pairing a wrist-mounted view with an overhead one. The resolution and frame rate examples are 640 by 480 at 30 fps, or 1920 by 1080 at 30 fps.

The rule that saves the most time is simple: if a human cannot tell from the image alone, neither can the model. The object hidden behind the wrist at the moment of grasp, the colour that shadow makes ambiguous, the depth you cannot judge from a single flat view. Those are layout problems to fix before recording starts. Discover them after fifty episodes and you have fifty useless episodes.

How many, and how varied

The official recommendation is at least 50 episodes, with 10 episodes per object location. The developers of the small model in question also recorded that 25 episodes was not enough and performance suffered — a rare and useful admission of a failed run.

Choosing the task itself matters as much as the count. A good first task satisfies three conditions:

It can be stated in one sentence
It finishes inside 60 seconds
Success or failure is objectively decidable by looking at the result

Being greedy with a long task means that when it does not work you cannot isolate which part failed.

The third practical condition is whether you can reset the object to the same place every time. The recommendation of ten episodes per location assumes the location genuinely stayed the same for those ten. If your reset is eyeballed, the ten episodes describe ten slightly different tasks, and the model has to average them.

Boring is the quality metric here

Demonstrations should be steady and unhesitating, which means the procedure should be settled before you start recording. Practise a few runs until your hand moves in the same order without thinking, then record. If you are going to record fifty, the ten practice runs pay for themselves easily.

Fatigue deserves planning too. A two-minute task repeated fifty times means the later demonstrations are not the same as the earlier ones. Breaking the session up and keeping the quality even is a better use of an afternoon than pushing through in one sitting.

The first fifty episodes are repetitive to the point of tedium. At this stage, the tedium is the quality indicator.

The counting trap when you resume

There is one trap that will cost you an hour if you meet it cold. When you stop recording and add more episodes later, the number you pass is how many to add, not the total you want to end up with. The documentation puts this in bold. Record 50, decide you want 80 in total, pass 80, and you now have 130.

Replay before you train

Before committing to a training run, replay an episode on the follower. It plays a recorded trajectory back through the hardware, which checks two things at once: that the recording is intact, and that the arm can physically reproduce it. A wiring mix-up or a calibration drift shows up here, in seconds, rather than after hours of training.

Where the data sits in the whole pipeline

It helps to remember why quality matters more than quantity here. Imitation learning is expected to converge in five to ten epochs over the dataset. There is no long grind in which a large model eventually averages out your inconsistencies. What you recorded is close to what you get.

That also means the fastest way to improve results is usually not more episodes. It is the same fifty, recorded better, on a task cut to a size the model can actually learn.

What to take away

Calibrate both arms first. Without it the dataset describes a slightly different robot than yours
Keyboard teleoperation needs X11, a Windows desktop, or macOS with permissions granted. Control-flow keys work anywhere
640 by 480 at 30 fps is the standard example. If a person cannot tell from the frame, the model cannot either
At least 50 episodes, 10 per object location. 25 was recorded as insufficient
One sentence, under 60 seconds, objectively judgeable — that is the shape of a first task
When resuming, the number you pass is episodes to add, not the target total
Replay one episode on hardware before training. It catches wiring and calibration faults in seconds

For how the arms are built and what they cost, see the SO-101 build. For what happens to this dataset next, see fine-tuning SmolVLA on your own GPU. The model landscape it feeds into is in robot foundation models in 2026.