From Jim Fan · NVIDIA GEAR
Three fuels for
embodied intelligence
Robots do not get an internet of actions. They must borrow common sense from human data, rehearse inside synthetic worlds, and pay for experience in physical reality.
The thesis
A capable robot foundation model will need all three data sources. Each one supplies what the other two are missing.
Jim Fan describes robotics data as a three-part portfolio: internet-scale data, simulation, and real-robot demonstrations. The useful distinction is not simply synthetic versus real. It is whether the data offers breadth, action labels, or physical truth. Today, no single source offers all three.
Web data
Watch the world before acting in it.
Human videos, images, text, and instructions expose models to a huge range of objects, activities, and environments. They teach semantic context and common-sense priors: what a mug is, how a drawer usually opens, or what tends to happen after water spills.
- Superpower
- Diversity at internet scale
- Blind spot
- No native motor commands
Seeing a hand move is not the same as knowing the joint positions, forces, or control signals that produced the motion.
Simulation
Practice faster than the clock.
A simulator provides the missing action-to-consequence loop. Policies can attempt a grasp, fail, recover, and repeat across thousands of parallel worlds. The marginal cost of another episode is low, and experience scales with compute.
- Superpower
- Abundant labeled interaction
- Blind spot
- The sim-to-real gap
Graphics, contact physics, friction, latency, sensor noise, and environmental diversity can all differ from reality.
Real-robot data
Learn where physics cannot be approximated.
Teleoperation and human demonstrations record what actually happens on a specific body: camera frames, robot state, actions, contact, timing, and failures. This is the strongest grounding data for imitation learning and deployment.
- Superpower
- Embodiment-specific reality
- Blind spot
- Human time and hardware cost
Collection is bound to wall-clock time, operators, resets, maintenance, safety procedures, and physical wear.
The trade space
Each source wins on a different axis.
| Data source | Breadth | Action signal | Physical fidelity | Marginal cost |
|---|---|---|---|---|
| Web | ||||
| Simulation | ||||
| Real robot |
My synthesis
The advantage is the mixing strategy.
The lesson is not to choose one source. It is to assign each source the job it does best, then create a loop in which real failures improve the simulator and targeted demonstrations fill the remaining gaps.
-
01
PretrainBroad priors from web data
-
02
RehearseAction-rich practice in simulation
-
03
GroundCalibrate with real demonstrations
-
04
Deploy & learnTurn failures into the next curriculum
Strategic implication
Manual teaching is expensive. That makes data operations strategic.
A hands-on training approach such as teleoperation can produce high-fidelity, task-relevant demonstrations with no sim-to-real gap. Its constraint is economics: every trajectory consumes operator and robot time.
The strategic question is therefore not only how to collect more demonstrations, but how to make each one compound: reuse across tasks, prioritize informative edge cases, automate quality checks, and combine demonstrations with simulation and pretrained visual-language representations.
Web data supplies common sense. Simulation supplies practice. Real robots supply ground truth.