ATLAS

ATLAS: Agentic Harness for Whole‑Body Intelligence from Human Video

Junli Ren*1,3, Kaixiang Yao*1, Junfeng Long1, Tao Huang3, Huayi Wang3, Yuxiao Chen2, Wenhao Ding2, Yan Wang2,
Wenjie Luo2, Boris Ivanovic2†, Marco Pavone2,4†, Masayoshi Tomizuka1†, Ping Luo3†, Yuxin Chen1^, Ran "Thomas" Tian1,2^

1UC Berkeley 2Autonomous System & Physical AI Research Group, NVIDIA 3The University of Hong Kong 4Stanford University

* co-first author  ·  † equal advising  ·  ^ project lead
The work was done at UC Berkeley, in academic collaboration with NVIDIA.

Best Demo Award · Agentic Robotics Workshop, CoRL 2026

“Go down the stairs, pick up the chip bag, and come back up.”1×
“Take a candy from the basket and put it in the pumpkin bucket.”2×
“Climb the stairs and pick up the paper ball.”climb 1×, rest faster
“Navigate to the sofa; pick up the trash along your way and put it in the bin.”1.5×

/ 01Overview

One Human Video Demonstration, One Prompt

From one human video demonstration and one prompt, with no robot demonstrations and no task-specific training in simulation, ATLAS completes the task the prompt names. An agentic harness, not a policy pretrained on robot data, links the agent and the whole-body tracker, and it is a part of each. It learns at test time what neither can know in advance: how this real robot executes a command, and the fine-grained task specification in this scene.

/ 02Method

An Agentic Harness

ATLAS divides the work by what each layer knows and when it learns it. System 2, the agent, knows the task before any session; System 0, the whole-body tracker, knows whole-body motion. The harness, a part of each, links the two and learns what only the running robot shows.

Top left, a person bending to open an under-counter cabinet door, face blurred, tagged Human video demonstration; below it, the robot crouching at the same cabinet with its end effector on the opening door, tagged with the prompt “Open the cabinet door.” An arrow marked 1, reference motion, leads from the person to a plate tagged System 2, the Agent, which knows the task; an arrow labelled tracks leads from the lower plate, System 0, frozen: the Whole-body tracker, which knows whole-body motion, to the robot. Between the plates, an arrow down reads edited motion and an arrow up reads attempts. A blue bracket joins the plates’ two blue bands to the ATLAS harness, built by the agent, learns at test time. (2) Phase segmentation, once, from the demonstration: chips end effector right and object “the silver metal cabinet door handle.”, steps walk and pick with the contact marked inside pick, and an orange loop: on a miss, retry the pick in place. (3) Motion editing, on every attempt: a dashed reference path and a blue edited path ending at the target n sub T, from the head camera, plus u sub t learned by the fast system.
ATLAS links the VLM agent and the frozen whole-body tracker with a harness, not a policy pretrained on robot data; the harness is a part of each layer. (1) The person’s retargeted motion is the reference. (2) From it and the prompt, the agent writes the task specification once: the steps, the end effector, the contact and the phrase the segmenter looks for. (3) On every attempt, the harness edits the reach to end at the target nT found in the head-camera view, and the frozen tracker follows it; the fast system learns a small correction ut, and a miss is retried in place. The frames show the cabinet-door task.

/ 03Learning

Learning on the Robot

The fast system starts from zero in each session and learns how this robot misses the target set by the agent and the camera; it never moves the target.

Learning throughout the session.

Every miss becomes training data.

Three attempts from one continuous stair run play alongside conceptual illustrations. The first two reaches miss the target and the robot retries in place; the learner keeps training on the attempts throughout the session, and the third reach grasps the object.

One autonomous learning session

Follow 38 attempts as the robot learns this task.

Pick up the trash and put it in the bin.

/ 04Tasks

One Harness, Multiple Tasks

Cabinet opening, box carrying, and object pickup use the same harness, with each task specified by one human video demonstration and one prompt.

Open the cabinet door. 1×

Pick up the box, carry it and set it down. 2×

/ 05Evaluation

Controlled Tests in Simulation

In Isaac Lab, online model-based RL is compared with a no-learner control, first with the real robot’s mean execution bias held fixed, then with the bias drawn anew for every attempt.

Online learning in simulation

Closing the execution gap.

Lower error. Higher success. Within about ten attempts.

  • Online model-based RL ours
  • No learner same bias, no residual

Apple

End-effector errorIn the hold · cm · median

↓ Lower is better

Success rate% · mean

↑ Higher is better

Banana

End-effector errorIn the hold · cm · median

↓ Lower is better

Success rate% · mean

↑ Higher is better

3 seeds · 80 attempts per seed · 10-attempt bins

Shaded bands: range across seeds. Gray strip: warm-up.

Online model-based RL against a no-learner control, in the same runs. Isaac Lab floor picks, one object per run, scored when the object is lifted. The real robot’s mean execution bias (7.9 cm short, 4.1 cm to the right) is held fixed and added to the end-effector command in every condition. After 5 warm-up attempts, a coin deals each attempt to the learner, to no residual (the control) or to exploration, which is not scored.

Ablation in simulation

A learned dynamics model, not a bias estimate.

When the execution bias changes from one attempt to the next, as it does on the real robot, the learned dynamics model lifts the apple in 83% of attempts; the best estimate of the bias, in 58%.

Attempts that lift the apple
Attempts that lift the appleAttempts that lift the apple
Online model-based RLours · learned dynamics
59 of 71 lifted · 95% interval 73–90%83%
Bias estimaterecursive least squares
66 of 114 lifted · 95% interval 49–67%58%
Model-free RLsame reward, no model
9 of 71 lifted · 95% interval 7–22%13%
No learnercontrol
16 of 378 lifted · 95% interval 3–7%4%

Isaac Lab · apple · 3 runs of 80 attempts per learner

Every learner starts from zero. The real robot's measured execution bias, drawn anew for every attempt with its measured spread, is added to the end-effector command; no method sees it. Bars count the attempts dealt to each learner after the 5 warm-up attempts; the control pools the uncorrected attempts of all runs. Apple shown; pooled with the banana as in the paper's table, 75% against 64%. Whiskers: 95% interval.

From demonstration to execution

Each three-panel strip compares the human video demonstration, direct motion retargeting, and ATLAS, aligned by phase.

Banana · from one human video demonstration
Apple · the same human video demonstration, another object
Box · carried with both arms
Fridge door · person and ATLAS, no retarget

/ 06Details

Setup & Scope

Setup

Which whole-body tracker is used?

The frozen whole-body tracker is ScaleBFM, trained before any session to follow human motion and adapt to terrain; it carries the robot up the stairs. The agent, the fast system and the retry are the same on the stairs as on the floor.

What runs on the robot, and what off it?

The robot senses with its own head camera (RGB-D) and joint state; some flat-ground tasks also localize it with an external tracking system. The whole-body tracker and the fast system's actor run on the robot; the VLM, SAM 3 and the learner's updates run on a PC over a link.

Does a person do anything during a run?

No one teleoperates the robot. A person only resets the object, after a success or when it leaves the camera's field of view, and steps in if the robot walks out of the workspace.

Scope

Next we will carry the fast system past the grasp, into sustained contact, and reuse real-robot experience so a new task starts from what the robot already knows.