ATLAS: Agentic Harness for Whole‑Body Intelligence from Human Video
1UC Berkeley 2Autonomous System & Physical AI Research Group, NVIDIA 3The University of Hong Kong 4Stanford University
* co-first author · † equal advising · ^ project lead
The work was done at UC Berkeley, in academic collaboration with NVIDIA.
Best Demo Award · Agentic Robotics Workshop, CoRL 2026
- Fully autonomous
- No teleoperation
- No policy switch
- No task-specific training in simulation
- No VLA / WAM
- All sensing onboard
/ 01Overview
One Human Video Demonstration, One Prompt
From one human video demonstration and one prompt, with no robot demonstrations and no task-specific training in simulation, ATLAS completes the task the prompt names. An agentic harness, not a policy pretrained on robot data, links the agent and the whole-body tracker, and it is a part of each. It learns at test time what neither can know in advance: how this real robot executes a command, and the fine-grained task specification in this scene.
/ 02Method
An Agentic Harness
ATLAS divides the work by what each layer knows and when it learns it. System 2, the agent, knows the task before any session; System 0, the whole-body tracker, knows whole-body motion. The harness, a part of each, links the two and learns what only the running robot shows.
/ 03Learning
Learning on the Robot
The fast system starts from zero in each session and learns how this robot misses the target set by the agent and the camera; it never moves the target.
Learning throughout the session.
Every miss becomes training data.
Three attempts from one continuous stair run play alongside conceptual illustrations. The first two reaches miss the target and the robot retries in place; the learner keeps training on the attempts throughout the session, and the third reach grasps the object.
One autonomous learning session
Follow 38 attempts as the robot learns this task.
Pick up the trash and put it in the bin.
- Success rate over the last eight attempts rises from 0% to 88% in 38 attempts.
- No teleoperation. A person only resets the trash: after a success, or when it leaves the camera's field of view.
/ 04Tasks
One Harness, Multiple Tasks
Cabinet opening, box carrying, and object pickup use the same harness, with each task specified by one human video demonstration and one prompt.
Open the cabinet door. 1×
Pick up the box, carry it and set it down. 2×
Pick up the box, carry it and set it down. 3×
Pick up the tennis ball. 1.5×
Pick up the can and put it in the bin. 3×
/ 05Evaluation
Controlled Tests in Simulation
In Isaac Lab, online model-based RL is compared with a no-learner control, first with the real robot’s mean execution bias held fixed, then with the bias drawn anew for every attempt.
Online learning in simulation
Closing the execution gap.
Lower error. Higher success. Within about ten attempts.
- Online model-based RL ours
- No learner same bias, no residual
Apple
End-effector errorIn the hold · cm · median
↓ Lower is betterSuccess rate% · mean
↑ Higher is betterBanana
End-effector errorIn the hold · cm · median
↓ Lower is betterSuccess rate% · mean
↑ Higher is betterShaded bands: range across seeds. Gray strip: warm-up.
Online model-based RL against a no-learner control, in the same runs. Isaac Lab floor picks, one object per run, scored when the object is lifted. The real robot’s mean execution bias (7.9 cm short, 4.1 cm to the right) is held fixed and added to the end-effector command in every condition. After 5 warm-up attempts, a coin deals each attempt to the learner, to no residual (the control) or to exploration, which is not scored.
Ablation in simulation
A learned dynamics model, not a bias estimate.
When the execution bias changes from one attempt to the next, as it does on the real robot, the learned dynamics model lifts the apple in 83% of attempts; the best estimate of the bias, in 58%.
Every learner starts from zero. The real robot's measured execution bias, drawn anew for every attempt with its measured spread, is added to the end-effector command; no method sees it. Bars count the attempts dealt to each learner after the 5 warm-up attempts; the control pools the uncorrected attempts of all runs. Apple shown; pooled with the banana as in the paper's table, 75% against 64%. Whiskers: 95% interval.
From demonstration to execution
Each three-panel strip compares the human video demonstration, direct motion retargeting, and ATLAS, aligned by phase.
/ 06Details
Setup & Scope
Setup
Which whole-body tracker is used?
The frozen whole-body tracker is ScaleBFM, trained before any session to follow human motion and adapt to terrain; it carries the robot up the stairs. The agent, the fast system and the retry are the same on the stairs as on the floor.
What runs on the robot, and what off it?
The robot senses with its own head camera (RGB-D) and joint state; some flat-ground tasks also localize it with an external tracking system. The whole-body tracker and the fast system's actor run on the robot; the VLM, SAM 3 and the learner's updates run on a PC over a link.
Does a person do anything during a run?
No one teleoperates the robot. A person only resets the object, after a success or when it leaves the camera's field of view, and steps in if the robot walks out of the workspace.
Scope
Next we will carry the fast system past the grasp, into sustained contact, and reuse real-robot experience so a new task starts from what the robot already knows.


