TL;DR

NVIDIA Cosmos world model development platform adds four new open-weight checkpoints to its Cosmos 3 model family for generating interactive world simulation and multi-sensor multi-view driving scenes:

• 🌦️ Cosmos3-Nano-Transfer-Auto : Generate multi-view driving data with varied weather, lighting, and time of day.
• 🚗 Cosmos3-Nano-Sim-Auto : Stream synchronized camera and LiDAR observations for driving simulation.
• 🎨 Cosmos3-Nano-Sim-Transfer : Turn simulator controls and a text prompt into photorealistic video.
• 🤖 Cosmos3-Nano-Sim-Bimanual : Predict video from robot or camera actions.

All four models are post-trained examples of Cosmos 3 Nano and have 16B trainable parameters. The models use OpenMDW1.1, permitting commercial and non-commercial use under its terms.

Download the checkpoints · Read the technical report . Watch the demos

Why do you need these checkpoints?

🦾 Robotics needs environments that can answer this across changing scenes and viewpoints. These checkpoints offer complementary ways to build them: Sim-Bimanual predicts visual consequences from actions, while Sim-Transfer turns a physics simulator’s controls into generated observations.

🚦 For autonomous driving, appearance and geometry must agree. Synchronized camera and LiDAR generation connects visual observations with 3D sensing. Scene and traffic-signal controls let researchers vary road layouts and signal states, then study how a driving policy responds.

🔄 Each observation can shape the next action. Streaming generation connects video prediction to ongoing interaction, giving world-model researchers a way to explore how actions and observations unfold together.

🌐 An open foundation for others to build on. NVIDIA’s vision is to help researchers and developers create richer, more responsive simulated worlds. Open weights and post-training recipes provide a shared starting point for adapting models, comparing methods, and contributing improvements. These problems remain hard: the work ahead is to measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short.

Build Autonomous Vehicle Worlds with Cosmos3-Nano

🌦️Transfer-Auto: driving scenarios → multi-sensor data

Cosmos3-Nano-Transfer-Auto is a bidirectional model for offline synthetic driving data generation. It generates synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB–LiDAR generation. Per-camera World Scenario Map (WSM) controls specify scene structure and traffic motion, while text prompts guide appearance.

Developers can vary environmental conditions to expand training and evaluation coverage, with optional initial RGB images to anchor scene appearance. Joint LiDAR generation additionally uses aligned HD-map range-map controls. Transfer-Auto also serves as the foundation for adapting Cosmos to your own sensor rig and driving domain. Start by post-training it on your data to accommodate your camera layout, LiDAR configuration, and target environment. The adapted model can be used directly for offline data generation or undergo causal training and few-step distillation for fast streaming simulation—the development path used to create Cosmos3-Nano-Sim-Auto.

Point your agent to the open skill : Run inference with Cosmos3-Nano-Transfer-Auto

Prefer a notebook? Open Transfer-Auto.

🚗Sim-Auto: driving controls → streaming observations

Cosmos3-Nano-Sim-Auto is the causally distilled streaming variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps, using generated history and updated driving controls to support fast streaming and long, interactive rollouts.

Provide per-camera captions and synchronized WSM controls, optionally adding initial RGB images. For LiDAR generation, include HD-Map range controls aligned with the camera controls. The distilled model generates successive chunks using its rolling history. Closed-loop interaction requires a simulation runtime and WSM renderer to update those controls from driving inputs. For a new sensor rig or domain, adapt Transfer-Auto first, then distill it into Sim-Auto.

Point your agent to the open skill : Run inference with Cosmos3-Nano-Sim-Auto

Prefer a notebook? Open Sim-Auto.

🎨 Sim-Transfer: simulator controls → RGB

Your simulator computes motion, contacts, and physical state. Sim-Transfer uses its spatial controls and a text prompt to render color observations that a policy can consume.

Action → simulator update → spatial controls → generated RGB → next policy action

• Depth: relative distance from the camera.

• Segmentation: regions and object boundaries.

• Edges: scene contours.

• Blur: coarse appearance and layout.

Text guides appearance; physics and ground-truth state stay with the simulator.

Read the full original article:

HuggingFace Blog