Skip to content

Cosmos-Transfer2.5 (Controlled Video Generation)

Overview

Cosmos-Transfer2.5 is a multi-controlnet model built on top of Cosmos-Predict2.5, producing high-quality videos conditioned on multiple spatial control inputs such as edge, depth, segmentation, and blur. With Cosmos-Transfer2.5, you can convert simulation videos into photorealistic ones (Sim2Real) or alter the background of real videos (Real2Real) to augment the training data for Physical AI. This tutorial demonstrates how to compile the Cosmos-Transfer2.5 pipeline and the edge ControlNet with multiple RBLN NPUs using RBLN Optimum (optimum-rbln), and generate a new video from a control video and a text prompt.

Info

Rebellions Scalable Design (RSD) is available on ATOM™+ (RBLN-CA22) and ATOM™-Max (RBLN-CA25). You can check the type of your current RBLN NPU using the rbln-stat command.

Setup & Installation

Before you begin, ensure that your system environment is properly configured and that all required packages are installed. This includes:

Note

  • RBLN Compiler requires an RBLN Portal account.
  • The commands above are intended for a default pip install on Debian-based Linux such as Ubuntu. For all other configurations, refer to the Installation Guide for the supported install matrix and the applicable commands.

Note

nvidia/Cosmos-Transfer2.5-2B on HuggingFace is a gated model. Once access is granted by agreeing to the NVIDIA Open Model License, you can log in using the hf auth (huggingface-cli) command as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ hf auth login

    _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|
    _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|
    _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|
    _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|
    _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|

    To login, `huggingface_hub` requires a token generated from https://huggingface.co/settings/tokens.
Enter your token (input will not be visible):

Running with the RBLN Optimum (optimum-rbln) library

This step demonstrates how to compile the Cosmos-Transfer2.5 model to run on multiple RBLN NPUs.

Model Compilation for Multiple NPUs

Cosmos-Transfer2.5 combines the Cosmos-Predict2.5 pipeline with a ControlNet for each control modality. The official HuggingFace Hub repository keeps the ControlNet weights on separate revisions, so load the desired ControlNet variant (one of edge, depth, seg, or blur) with the AutoModel class of diffusers first, and pass it to the pipeline.

Then, compile models in the Cosmos-Transfer2.5 pipeline using the RBLNCosmos2_5_TransferPipeline class provided by optimum-rbln. To compile the models, you must set the export argument to True. The class's from_pretrained() method then downloads the original models from the HuggingFace Hub and compiles them using the RBLN Compiler. After compilation, save the model artifacts to disk using the save_pretrained() method. This creates a directory (e.g., rbln-cosmos-transfer2.5) containing the compiled model.

Before compiling the models, specify the shape of the output video. Long videos are generated auto-regressively in num_frames-sized chunks, so the compiled shapes are per chunk (93 frames, 704x1280 pixels) while the total output length follows the length of the input control video.

In addition, several modules in the pipeline need multiple NPUs for inference, so specify the number of NPUs at compile time. The example code for compilation is as follows.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import torch
from diffusers import AutoModel
from optimum.rbln import RBLNCosmos2_5_TransferPipeline

# Define the HuggingFace model ID
model_id = "nvidia/Cosmos-Transfer2.5-2B"

# Load the edge ControlNet from its own revision
controlnet = AutoModel.from_pretrained(
    model_id,
    revision="diffusers/controlnet/general/edge",
    torch_dtype=torch.bfloat16,
)

# Compile Cosmos core models (transformer, controlnet, text_encoder, vae, safety_checker)
pipe = RBLNCosmos2_5_TransferPipeline.from_pretrained(
    model_id,
    revision="diffusers/general",
    controlnet=controlnet,
    torch_dtype=torch.bfloat16,
    export=True,
    rbln_config={
        "height": 704,
        "width": 1280,
        "num_frames": 93,
        "max_seq_len": 512,
        "create_runtimes": False,
        "text_encoder": {
            "num_devices": 8,
        },
        "transformer": {
            "num_devices": 8,
        },
        "controlnet": {
            "num_devices": 2,
        },
    },
)

pipe.save_pretrained("rbln-cosmos-transfer2.5")

Load the Compiled RBLN Model

To generate videos, load the compiled models saved in the previous step. You can load them with the from_pretrained() method of the RBLNCosmos2_5_TransferPipeline class. In this step, set the export argument to False to indicate that you are loading precompiled models from a local path.

The Cosmos-Transfer2.5 pipeline consists of many modules, some of which occupy a large amount of device memory. Loading all of them on the same device causes a device out-of-memory error. Even with the same number of NPUs, loading can succeed or fail depending on which modules are placed on which devices, so set the device IDs in rbln_config carefully. In the example code below, every submodule is placed within the 8 NPUs that the transformer spans. text_encoder uses all eight, and the controlnet, the VAE encoder/decoder (which also encodes the control video), and the safety_checker are placed on the first four, so the whole pipeline fits in one pool of 8 NPUs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
from optimum.rbln import RBLNCosmos2_5_TransferPipeline

pipe = RBLNCosmos2_5_TransferPipeline.from_pretrained(
    model_id="rbln-cosmos-transfer2.5",
    export=False,
    rbln_config={
        "text_encoder": {
            "device": [0, 1, 2, 3, 4, 5, 6, 7],
        },
        "transformer": {
            "device": [0, 1, 2, 3, 4, 5, 6, 7],
        },
        "controlnet": {
            "device": [0, 1],
        },
        "vae": {
            "device_map": {"encoder": 2, "decoder": 3},
        },
        "safety_checker": {
            "qwen3guard": {"device": 0},
            "face_blur_filter": {"device": 0},
        },
    },
)

Video Generation

After the compiled pipeline is loaded, generate videos as follows. Call the pipeline with your control video and prompt to generate a new video that follows the structure of the control video (edge maps). The negative prompt defaults to the value defined inside the pipeline, and you can pass your own prompt through the negative_prompt argument. The total length of the output video follows the control video (121 frames in the example below) by default, and the video is generated in the 93-frame chunks specified at compilation.

Note

While the original cosmos-transfer2.5 implementation supports multi-control, which combines multiple control modalities at once, the current diffusers implementation (as of version 0.38.0) accepts only a single ControlNet and a single control video. To use a different control modality, recompile the pipeline with the corresponding ControlNet (depth, seg, or blur).

NVIDIA Open Model License

In accordance with the NVIDIA Open Model License, the safety_checker submodule runs the guardrail over the input prompt and the generated video automatically inside the pipeline. Do not remove or disable it. Under the license, bypassing the guardrail or reducing its efficacy terminates your rights automatically, unless you substitute a substantially similar guardrail appropriate for your use case.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import torch
from diffusers.utils import export_to_video, load_video

# Control video: the pre-computed Canny edge maps for the car driving scene from the
# official cosmos-transfer2.5 assets.
controls = [
    load_video(
        "https://github.com/nvidia-cosmos/cosmos-transfer2.5/raw/refs/heads/main/assets/car_example/edge/car_edge.mp4"
    )
]
prompt = "The video is a driving scene through a modern urban environment, likely captured from a dashcam or a similar fixed camera setup inside a vehicle. The scene unfolds on a wide, multi-lane road flanked by tall, modern buildings with glass facades. The road is relatively empty, with only a few cars visible, including a black car directly ahead of the camera, maintaining a steady pace. The camera remains static, providing a consistent view of the road and surroundings as the vehicle moves forward.On the left side of the road, there are several trees lining the sidewalk, providing a touch of greenery amidst the urban setting. Pedestrians are visible on the sidewalks, some walking leisurely, while others stand near the buildings. The buildings are a mix of architectural styles, with some featuring large glass windows and others having more traditional concrete exteriors. A few commercial signs and logos are visible on the buildings, indicating the presence of businesses and offices.Traffic cones are placed on the road ahead, suggesting some form of roadwork or lane closure, guiding the vehicles to merge or change lanes. The road markings are clear, with white arrows indicating the direction of travel. The sky is clear, suggesting a sunny day, which enhances the visibility of the scene. Throughout the video, the vehicle maintains a steady speed, and the camera captures the gradual approach towards the intersection, where the road splits into different directions. The overall atmosphere is calm and orderly, typical of a city during non-peak hours."

# Transfer inference with controls: the output video follows the edge maps.
video = pipe(
    controls=controls,
    controls_conditioning_scale=1.0,
    prompt=prompt,
    height=704,
    width=1280,
    generator=torch.Generator().manual_seed(1),
).frames[0]
export_to_video(video, "generated_video.mp4", fps=16)

Example Output:

The full example code is available in the RBLN Model Zoo GitHub repository.

References