Skip to content

Collectives across cards with rbln-ccl

On Rebel100™, one card is one logical device, and work that spans several cards runs as several processes that exchange tensors through torch.distributed. RBLN PyTorch provides the rbln-ccl backend for that: the collectives run on the NPUs and the script is the same one you would write for any other accelerator. This tutorial launches two processes on two cards and assembles a table with all_gather.

How a card is partitioned internally is not part of this picture. A Rebel100™ card is a fixed group of four chiplets that the RBLN Compiler uses for every program on its own; see the RBLN Compiler documentation. The device-mapping variables that group ATOM™ cards into one logical device are not used on Rebel100™; see Environment Variables › Device mapping.

Prerequisites

  • A host with at least two RBLN NPUs and the RBLN Driver installed. The output below was recorded on two Rebel100™ (RBLN-CR13) cards.
  • Python: 3.10–3.14
  • torch, torch-rbln, and rebel-compiler installed as described in Installation. torchrun ships with torch.

Step 1: Gather one row from every card

ccl_allgather.py is a standard torch.distributed script with one RBLN-specific line: each rank picks its own card from the RBLN_VISIBLE_DEVICES list before the first device use, because torchrun gives every rank the same environment. Each rank then fills a small tensor with its rank number and all_gather assembles the full table on every rank.

ccl_allgather.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import os

import torch
import torch.distributed as dist

# torchrun gives every rank the same environment. Pick this rank's card from
# the RBLN_VISIBLE_DEVICES list before the first device use.
local_rank = int(os.environ["LOCAL_RANK"])
os.environ["RBLN_VISIBLE_DEVICES"] = os.environ["RBLN_VISIBLE_DEVICES"].split(",")[
    local_rank
]

dist.init_process_group(backend="rbln-ccl")
rank, world_size = dist.get_rank(), dist.get_world_size()

# Each rank contributes one row; all_gather assembles the full table on every rank.
mine = torch.full((4,), float(rank), dtype=torch.float16, device="rbln")
gathered = [torch.empty_like(mine) for _ in range(world_size)]
dist.all_gather(gathered, mine)

table = torch.stack(gathered).cpu()
print(f"rank {rank} sees {tuple(table.shape)}: {table[:, 0].tolist()}")
dist.destroy_process_group()

$ RBLN_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node 2 ccl_allgather.py
rank 0 sees (2, 4): [0.0, 1.0]
rank 1 sees (2, 4): [0.0, 1.0]

torchrun prints its usual note about setting OMP_NUM_THREADS first, and the two lines may arrive in either order.

How to read it:

  • Every rank ends with the same table. Row 0 came from rank 0 and row 1 from rank 1; after all_gather, both processes hold both rows. The gather ran on the NPUs because the tensors are float16; Execution Semantics › Distributed lists how other dtypes are handled.
  • One process, one card. RBLN_VISIBLE_DEVICES=0,1 names the two cards for torchrun, and the first lines of the script narrow it to one card per rank. The mapping is committed at the first device use, so this has to happen before any tensor is created on rbln.
  • torchrun provides the rank information. init_process_group reads the rendezvous variables torchrun sets. The runtime's own control plane defaults to 127.0.0.1, which is enough for one host.
  • The call returns when the collective is complete. rbln-ccl collectives are synchronous by default; TORCH_RBLN_C10D_ASYNC=1 returns work handles to wait on instead.

Step 2: Go beyond one host

The same script runs across hosts once two things change. Launch it with torchrun --nnodes and a rendezvous endpoint, as for any backend, and set the RBLN control plane on every rank before init_process_group: RBLN_ROOT_IP to rank 0's address, RBLN_LOCAL_IP to the rank's own address, and RBLN_RDMA_HCA (or RBLN_RDMA_IP) when a host has more than one RDMA interface. The variables and their defaults are in Environment Variables › Distributed.

The supported collectives (broadcast, scatter, all_gather, all_reduce, reduce_scatter, send, recv, barrier), the reduce operators, and the dtype rules are listed in Execution Semantics › Distributed.

Next steps