World models: how dynamics prediction changes robot policy

The problem WAMs solve

In robotics, training a policy that works only on demonstration scenes is easy. Making it work when objects change shape, position, or lighting is another story. A model trained on a white table fails on a wooden one. A gripper that learned to close on a cube does not close the same way on a sphere.

The standard approach so far has been to add an action module to a pretrained vision-language model, creating a vision-language-action (VLA) model. It works well for understanding natural-language instructions and recognizing objects. But here is the problem: a VLM learns to describe the world, not to predict how it evolves. That missing dynamics model is exactly what a robot needs when the task depends on anticipating how a scene will change. When the gripper closes, what happens to the cup? When an object is released, where does it fall?

NVIDIA proposes replacing the language backbone with a video-based world model, creating what it calls a World Action Model (WAM). Jim Fan, a researcher at NVIDIA, summed up this transition in one line: "VLAs are dead, long live World Action Models."

What changes in practice

The difference is not theoretical. A VLM is optimized to generate text about images. WAMs model how a scene evolves. That means during post-training you do not have to teach dynamics from scratch: the policy specializes on top of a model that already has a prior understanding of physics.

According to NVIDIA’s paper "World Action Models are Zero-shot Policies", this translates into three practical gains: you need less data to reach a given capability, behavior generalizes better out of the training distribution, and the path to a new platform (another robot, another gripper) is shorter. These properties come from the pretraining backbone itself, so they show up in any policy that specializes from it.

Cosmos 3, the foundation

NVIDIA presents Cosmos 3, an omnimodal world model based on a Mixture-of-Transformers architecture. The multimodal input stream passes through an autoregressive transformer that generates discrete tokens like text. That guides a diffusion transformer for continuous modalities: image, video, audio, and action. Text is generated by token-by-token decoding; everything else, including actions, is synthesized by iterative denoising.

Cosmos 3 comes in three sizes: 4B (Cosmos Edge), 16B (Cosmos Nano), and 64B (Cosmos 3 Super). What makes it solid as a specialization base is the breadth of its physical-world data: roughly 767 million images, 348 million videos of real dynamics, and 8 million action samples spanning robotic manipulation, autonomous driving, camera motion, and egocentric motion.

NVIDIA already offers post-trained policies: Cosmos3-Nano-Policy-DROID (16B) for the Franka Panda arm and Cosmos3-Edge-Policy-DROID (4B) for on-device deployment. Given a language instruction and multi-camera observations, both generate action trajectories for the robot.

A paradigm shift

WAMs represent a conceptual move: from "learning to act" to "learning how the world evolves". The robot does not memorize demonstrations; it anticipates consequences. Cosmos 3 makes this approach practical by offering three sizes that fit different deployment tiers without forcing a single trade-off.

Where conversational hardware comes in

At bitbitbla, we work on translating voice into deterministic JSON commands that hardware executes directly. The platform sends the detected intent, confidence, and action parameters over serial port or webhooks, ready for physical execution. The microcontroller acts as a terminal that receives direct orders, and context triggers let the agent react to environment variables in real time.

Instead of loading the microcontroller with heavy physics or vision models, at BitBitBla we keep device firmware light and deterministic. Contextual intelligence lives in our cloud orchestrator, which processes environment dynamics and sends the microcontroller the precise order in JSON over serial for immediate execution.

Source: NVIDIA Embedded.