RoboCap-10k

Read the technical report (PDF) · Download the dataset

Egocentric video is the most scalable source of manipulation data for robot learning, but video alone is not supervision. A robot imitates poses, not pixels, so every hour of footage becomes useful only once someone estimates where the head and hands were, in meters, at every frame. The largest egocentric corpora today are monocular, where metric scale can only be guessed from learned priors. The most accurate ones are captured in motion-capture labs with a handful of participants. Federated efforts pool heterogeneous devices and inherit their differences, and headset-labeled datasets inherit whatever a real-time tracker produced, with no raw multi-view data to re-estimate from.

Today we are releasing RoboCap-10k: 10,000 curated hours of unscripted, bimanual manipulation, recorded by workers during their paid shifts in 11 countries, across 21 industry verticals, 365 workplace types, and 619 distinct tasks. Every hour was captured on one device, RoboCap, and processed by one 3D stack, Grounded API. Every frame carries a gravity-aligned 6-DoF head pose and world-frame MANO hand poses, and every episode carries two tiers of language. We release the raw synchronized streams alongside the labels, so they can be audited, and regenerated as methods improve.

Real, in-the-wild, occupational work

Most egocentric datasets record household chores or tabletop tasks performed for the camera. RoboCap-10k records occupational work: bread and pastry production, industrial garment stitching, engine and motorcycle repair, circuit-board soldering, rebar tying, onion harvesting. Wearers are existing employees recording during their regular 6–12 hour shifts; the hat is the only change to their work, and nothing is scripted, staged, or repeated for the camera. Policies degrade sharply under visual shift, so the clutter, lighting, and tools of each workplace are left exactly as they are.

These are curated hours, not raw recording time. An episode is one continuous clip in which a person performs one recognizable task with both hands tracked and SLAM available, and it is kept only if it passes calibration, SLAM, and hand-tracking checks. The corpus holds 1,363,915 episodes, with a median length of 17.8 s, about a billion synchronized timesteps, and 6.5 billion camera frames across six streams.

Every task is labeled with the industry vertical it was recorded in and, independently, the skill group it requires, and is rated by difficulty. Easy tasks are single-step and need no training, like sorting groundnuts. Medium tasks need coordination or planning, like shaping dough into loaves. Hard tasks are skilled work that takes training or expertise, like soldering components onto PCB panels, weaving cane onto chair frames, or stitching garments on industrial machines; 172 of the 619 tasks are hard. The distribution is long-tailed by design, and because one label often covers several workflows, these counts are a floor on the true variety.

Action labels with sub-centimeter precision

Head trajectories come from Grounded API's multi-camera visual-inertial SLAM, refined offline with a keyframe bundle adjustment over the whole session once every revisit is known. Hand poses come from a feed-forward multi-view tracker that takes crops from every camera that sees a hand and predicts a single MANO pose consistent across views by construction. It holds up through gloves, tool contact, partial occlusion, clutter, and outdoor light.

We evaluate both against optical motion capture on RoboMoCap, recorded on RoboCap itself. Over 153.4 hours and 4,669 trajectories, head pose reaches a mean absolute trajectory error of 3.19 mm, with metric scale recovered to a median 0.17% without any scale alignment. On held-out test sequences, hands reach 4.5 mm PA-MPJPE over 21 keypoints, and on our internal benchmark, where the true hand shape is known, they reach 9.2 mm of absolute error (MPJPE). On the city-scale LaMAria benchmark, our SLAM scores 72.4, close to Aria's own SLAM at 75.9.

Standardized hardware, calibrated online

Because every hour comes from the same hardware design, there is no cross-device domain shift in either the imagery or the quality of the 3D labels. RoboCap is a 250 g hat with six hardware-synchronized global-shutter cameras: two fisheye stereo pairs that keep the hands inside a metric stereo frustum across the reach envelope, and two lateral wide-angle cameras that catch reaches leaving it. Two IMUs at 200 Hz share the cameras' clock.

Factory calibration is where accuracy starts, not where it ends. Wearable rigs deform under mechanical and thermal stress, and across our fleet the rotation between stereo cameras drifts by about half a degree, enough for a 1–2 cm depth bias at manipulation range. So for every recording session we re-estimate the rotations of all six cameras jointly from natural correspondences in the footage, holding the factory translations fixed so metric scale stays anchored. Across the release, every one of the 216 devices averages below 1 px of epipolar error, with a median of 0.63 px on the front pair and 0.60 px on the eye pair. Two devices that could not get below 1 px were excluded, removing 34 hours.

Task and subtask captioning

Manipulation at work is long-horizon, so every interval carries one free-text sparse caption describing its overall intent, and is cut into consecutive dense units describing what happens at each moment. Each dense unit records the scene, the sub-task it belongs to, and the physical action, which together supervise both task planning and short-horizon control. The two tiers are generated independently, and captions for 1,000 hours were reviewed by people against the source video.

A new standard for egocentric data

RoboCap-10k is the only released dataset that combines calibrated stereo, metric head pose, and 3D hands with more than 10,000 hours of occupational work.

Ready to train, safe to share

Every episode ships in two formats. The LeRobot format is for training: each camera is an H.265 MP4, and per-frame head pose, MANO parameters and keypoints for both hands, and IMU readings are stored in Parquet tables indexed by frame, so the corpus drops into existing dataloaders. A Rerun recording of the same episode is for inspection, placing the six streams, head trajectory, and hand meshes in one 3D world frame on a common timeline. Each episode also carries its factory and online-corrected calibration and a JSON sidecar with taxonomy labels, captions, and per-session quality statistics. The dataset is released under CC BY 4.0, open to research and commercial use alike.

Every operator signed an informed consent form, every partner enterprise signed a data agreement, and wearers are compensated on top of their wages. No audio is recorded, and released metadata identifies only the country and site type. Off-the-shelf face detectors miss many faces in this footage, seen from above, through fisheye optics, at the distorted periphery, so we trained our own on roughly 1,000 hours of labeled RoboCap footage. At the released operating point it recalls 77.3% of faces at 92.4% precision, against 25.1% for RetinaFace, and detections are tracked and propagated across frames so a face missed in one frame is still covered by its neighbors. Blurring is irreversible; the originals are not distributed.

We release RoboCap-10k as pretraining data for generalist manipulation policies and world models, and as a testbed for a question the field has not yet answered: how much does the precision of human action labels matter at scale?