Skip to content

Cosmos WFMs (World Foundation Models)

Overview

Cosmos is a platform for accelerating physical AI development. Cosmos WFMs (World Foundation Models) are neural networks that can predict and generate physics-aware videos of the future state of a virtual environment. WFMs generate and simulate virtual worlds with text, image, video and movement in a way that accurately models the spatial relationships of objects in the scene and their physical interactions.

Cosmos WFMs consist of several independent AI subsystems, and every one of them runs on the Rebellions ecosystem. Cosmos WFMs come in three model types, and the tables below list each generation of a type along with a link to its tutorial. The tutorials are sequentially connected, but each of them can be tested independently.

The figure below shows Text-to-World inference with Cosmos-Predict1-7B on an RBLN NPU.

Two rows, each pairing a boxed input prompt with an arrow to the video generated from it Two rows, each pairing a boxed input prompt with an arrow to the video generated from it

Cosmos WFMs Components

Cosmos-Predict: World Generation Models

The Predict models predict and generate physics-aware images and videos. Cosmos-Predict2.5 improves on the earlier generations by unifying Text2World, Image2World, and Video2World into a single flow-based model that uses Cosmos-Reason1 as the text encoder.

Model Size Capability Input
Cosmos-Predict1 7B, 14B Text2World, Video2World text/image/video
Cosmos-Predict2 2B, 14B Text2Image, Video2World text/image/video
Cosmos-Predict2.5 2B, 14B Text2World, Image2World, Video2World text/image/video

Cosmos-Transfer: Multi-ControlNet Models

The Transfer models condition world generation on spatial control inputs such as edge, depth, segmentation, and blur. Cosmos-Transfer2.5 is built on top of Cosmos-Predict2.5 and produces world simulation videos that follow the control signal.

Model Size Capability Input
Cosmos-Transfer1 7B Multi-controlnet video generation text/control video
Cosmos-Transfer2.5 2B Multi-controlnet video generation text/control video

Note

diffusers has no Cosmos-Transfer1 pipeline, so the RBLN Model Zoo implements the model on NVIDIA's own codebase instead of the pipeline API the other Cosmos examples use. The runnable example is in RBLN Model Zoo - Cosmos-Transfer1.

Cosmos-Reason: Reasoning Vision-Language Models

The Reason models understand and reason about space, time, and physics in image and video inputs, answering with chain-of-thought reasoning. Cosmos-Reason2 is post-trained from Qwen3-VL-Instruct, and adds 2D/3D point localization with bounding-box coordinates and input context up to 256K tokens.

Model Size Capability Input
Cosmos-Reason1 7B Visual reasoning text/image/video
Cosmos-Reason2 2B, 8B, 32B Visual reasoning, object localization text/image/video

References