Execution Semantics¶

Chiplets and memory statistics¶

How the compiler splits a program across the four chiplets is described in the RBLN Compiler documentation; RBLN PyTorch exposes the chiplets only through the memory statistics. On Rebel100™ the caching allocator tracks each chiplet separately. memory_allocated(), memory_reserved(), and memory_stats() sum the chiplets, so an imbalance is invisible in them; memory_summary() prints one row per NPU and chiplet, and memory_stats_per_chiplet() returns the same keys as memory_stats() prefixed with npu.<n>.chiplet.<c>.. Call print(torch.rbln.memory_summary()) in the same Python process that ran the operation to inspect per-chiplet memory usage. See API and supported operations › Memory for the related APIs. The following output was captured after one float16 matrix multiply on a single card:

torch_rbln memory summary (device=rbln:0, MiB)
scope: pid 461977 -- caching allocator only, this process only
npu: physical NPU id as in device_summary() (relative to RBLN_VISIBLE_DEVICES, may differ from rbln-smi)
  npu  chiplet    allocated   alloc peak     reserved    resv peak       active       cached
--------------------------------------------------------------------------------------------
    0        0          1.2          1.7          2.0          2.0          1.2          0.8
    0        1          0.0          0.0          2.0          2.0          0.0          2.0
    0        2          0.0          0.0          2.0          2.0          0.0          2.0
    0        3          0.0          0.0          2.0          2.0          0.0          2.0
--------------------------------------------------------------------------------------------
         total          1.3          1.8          8.0          8.0          1.3          6.7
alloc retries: 0   ooms: 0

Peaks are tracked per chiplet, so the peak columns of the total row bound the joint peak from above; memory_stats() carries the exact joint peak. An out-of-memory error on a device whose aggregate statistics look comfortable is the signature of one full chiplet, and this table is where to look.

Alignment rules measured on Rebel100™¶

Two eager-dispatch rules decide whether a compute operator reaches the device at all. They are implemented in the dispatch layer rather than in hardware, so they apply on ATOM™ as well; the measurements in this documentation were taken on Rebel100™.

• An eager compute operator whose tensor argument has a last dimension that is not a multiple of 64 runs on the CPU. The fallback log names the operator, and explain() shows it as a repeating dispatch/recompile. A 3-element add, a Linear(128, 8) output, and the 32-wide half-head slices of a rotary embedding all trip it; padding the last dimension or compiling the block removes it.
• The fused attention kernel takes query sequence lengths that are multiples of 32. Other lengths fall back with sdpa_overrideable (query sequence length N not aligned to 32) op ran on CPU, which in eager generation means every single-token decode step.

Both rules are per operator and disappear inside a torch.compile graph, where the compiler pads and lowers on its own; see Pre-dispatch checks for the full check order.

Logical devices and RSD groups¶

A physical device is an NPU. A logical device is what PyTorch selects with an index such as rbln:N. By default the two map 1:1. With Rebellions Scalable Design (RSD), a logical device groups 2, 4, 8, 16, or 32 physical NPUs, and a compiled program running on it is partitioned across the group. On ATOM™ that grouping is configured with RBLN_DEVICE_MAP or RBLN_NPUS_PER_DEVICE, and the visible pool with RBLN_VISIBLE_DEVICES; see Environment Variables › Device mapping. On Rebel100™ the RSD group is fixed inside the card: the four chiplets form an RSD-4 group, every program is partitioned across them without configuration, and a logical device is always one card. Work that spans several Rebel100™ cards uses torch.distributed with rbln-ccl (Distributed).

On ATOM™, grouping several NPUs into one logical device is how tensor-parallel execution is configured. The group defines the range of NPUs an operator may use; how many it actually requests is decided per execution mode, described in Tensor-parallel execution. The rest of this subsection describes that ATOM™ configuration; on Rebel100™ leave the grouping variables unset.

For example, on a host with four ATOM™ NPUs the following groups them in pairs:

1

RBLN_NPUS_PER_DEVICE=2 python application.py

The application then selects rbln:0 or rbln:1; device_count() returns 2 and physical_device_count() returns 4.

The mapping follows these rules:

• The NPUs a process may see are limited by the runtime's RBLN_VISIBLE_DEVICES (RBLN_DEVICES is a deprecated alias). Physical NPU numbers in the grouping variables and in device_summary() are indices into that visible list.
• Set the grouping variables before the first real device use. Queries such as is_available() and device_count() only plan the mapping and claim no NPU. The mapping is committed at the first real use: an allocation, synchronize(), or a collective. set_device() is bookkeeping and does not commit. After commit, changes to the variables are ignored and the mapping stays fixed for the process lifetime. torch.cuda.device_count() behaves the same way: it caches its result only after CUDA is initialized.
• RBLN_DEVICE_MAP names the NPUs of each group explicitly and takes precedence over RBLN_NPUS_PER_DEVICE.

Unlike CUDA, where a multi-GPU model is split by the user with torch.distributed, an RSD group is a single logical device: a compiled graph is partitioned across the group by the RBLN Compiler, and eager code sees one device.

Selecting a device¶

To use a specific logical device, name its index, as in torch.device("rbln:N"). "rbln" without an index means the currently selected RBLN device, and device-agnostic code that uses torch.accelerator resolves to rbln when it is the active accelerator. The torch.rbln.device() context manager changes the selection inside its block only:

1
2
3
4
5
6
7
8
9

import torch

with torch.rbln.device(0):
    x = torch.tensor([1.0, 2.0], dtype=torch.float16, device="rbln")
    y = torch.tensor([3.0, 4.0], dtype=torch.float16, device="rbln")
    z = x + y

assert z.device == torch.device("rbln:0")
print(z.cpu())

tensor([4., 6.], dtype=torch.float16)

x, y, and z are all placed on rbln:0. Tensor creation inside the context still names device="rbln"; the context does not change the default device of factory functions and does not move tensors that already exist. Move an existing tensor with .to(target_device), and keep the tensors of one operation on the same logical device. A copy between two RBLN logical devices is handed to the runtime and may travel through host memory.

Processes and fork()¶

Do not fork() a process that has already used a device. The child inherits the parent's device state but not a usable runtime context, and unlike CUDA, which raises when a forked child re-initializes the device, this backend has no bad-fork detection: the child neither errors nor works reliably. A worker forked before any device use in the parent may still set the device-mapping variables and then use a device on its own. Prefer the spawn start method for multiprocessing, and keep DataLoader workers off rbln tensors.

Lazy device tensors¶

Creating a tensor on rbln does not immediately consume device memory. The runtime registers a virtual storage for the tensor and commits physical memory according to the allocation mode:

Keep the default for applications; the eager mode exists for experiments and profiling of allocation behavior. Set the variable before the process starts.

Under lazy allocation, the runtime tracks for every storage whether host memory or NPU memory holds the latest data and performs a transfer only when a consumer needs the other side:

This has four visible consequences:

• Memory statistics lag creation. torch.rbln.memory_allocated() counts committed device memory, so it reads low right after allocation and rises after the first device operation. Size device memory by running a representative input, not by summing tensor sizes, and expect that an out-of-memory error can surface at the first device operation on a tensor rather than at the torch.empty that reserved it. See Memory statistics and the caching allocator.
• Host-only workloads never touch the NPU. A script that creates rbln tensors and then only prints them, indexes them with Python integers, or runs float32 arithmetic on them executes entirely on the host. The tensors report device='rbln:0' throughout. A test written this way does not exercise device behavior.
• data_ptr() is a virtual address. It cannot be dereferenced from the CPU; read data with .cpu().

• External libraries that bypass PyTorch operators cannot access NPU physical memory that has not yet been allocated. External code that accesses NPU memory without going through PyTorch operators requires a buffer with physical memory allocated. With lazy allocation, creating a tensor may not allocate its physical memory, so call torch.rbln.bind_device_memory(tensor) before passing the buffer to that code. This function immediately allocates physical memory as one flat region for a contiguous tensor that covers its whole storage, with storage offset zero.

  To match another tensor's device memory layout, use torch.rbln.set_device_layout_like(target, ref). This configures the layout so later device-to-device copies use the fast path; it does not copy data. Both tensors must have the same device and dtype and cover their whole storage, and ref must be device-resident. For details, see API and supported operations › Memory.

Memory statistics and the caching allocator¶

Device memory is managed by a caching allocator. Like the CUDA caching allocator, it keeps freed memory blocks for reuse. These blocks are included in memory_reserved(), and torch.rbln.empty_cache() returns them to the runtime. empty_cache() also drops the compiled-program warm cache and view-recipe caches, so it releases everything the caller no longer holds; the next eager call pays its preparation cost again. Do not empty the cache between iterations of a loop whose performance you care about.

All of them report device memory only. Two effects make the numbers lower than a CUDA user expects: lazy tensors do not count until a device operation materializes them, and compiled programs cached by Dynamo hold their own device buffers, which show up as reserved memory that no live tensor accounts for. The following example reads the statistics after placement and after execution, with TORCH_RBLN_EAGER_MALLOC unset:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

import torch

device = "rbln:0"
x = torch.randn(128, 128, dtype=torch.float16, device="cpu").to(device)
print("Allocated after placement:", torch.rbln.memory_allocated(device))
torch.rbln.reset_peak_memory_stats(device)

with torch.inference_mode():
    result = x + x
    host_result = result.cpu()

print("Allocated after execution:", torch.rbln.memory_allocated(device))
print("Reserved after execution:", torch.rbln.memory_reserved(device))
print("Peak allocated during execution:", torch.rbln.max_memory_allocated(device))

Allocated after placement: 0
Allocated after execution: 233472
Reserved after execution: 8388608
Peak allocated during execution: 233472

The values depend on the SoC, the allocation granularity, and the cache state. Device statistics reflect the device layout, the allocation granularity, and runtime buffers, and exclude host process memory and tensor data that lives only on the host, so the sum of numel() * element_size() over your tensors and the reported device memory are different numbers. To measure tensor memory alone, materialize first and reset the Dynamo cache with torch._dynamo.reset() before reading. reset_peak_memory_stats() and reset_accumulated_memory_stats() behave as on CUDA.

Host memory: pinned and huge buffers¶

pin_memory=True is honored while RBLN is the active accelerator: pinned CPU tensors come from a page-aligned host allocator with best-effort mlock. Pinned memory is what makes non_blocking=True copies asynchronous (see Non-blocking copies).

Two helpers cover large host buffers: torch.rbln.huge_host_empty(nbytes) returns a 2 MiB-aligned, prefaulted buffer the device can DMA into without a staging copy, and torch.rbln.offload() pages the host-side regions of tensors allocated inside its block out to files. Both are described in APIs › Memory.

Operator dispatch path¶

For a tensor on rbln, the PyTorch dispatcher routes an ATen operator to the PrivateUse1 key, which RBLN PyTorch owns. What happens next depends on the operator's registration:

The compiled-kernel path has two cache levels. The first miss for an operator and input profile builds the program (seconds). Later dispatches with a matching profile hit a warm cache in C++ that drives the runtime directly, without entering Python or Dynamo, so steady-state eager overhead per operator is small. The cache key is the operator name plus, for every input, its dtype, shape, strides, storage offset, and device index, plus the values of arguments passed as Python scalars. A different value in any of these is a new program.

Some calls never reach the C++ warm cache even though they do not recompile: every call re-enters the Python wrapper and hits its compile cache there, which costs Python dispatch on each call but no compiler work.

• An input that is a recognized view (a permute, expand, narrow, select, or reshape of a contiguous base). The view chain is replayed inside the compiled wrapper, and the C++ key built from the view's shape does not match the program compiled for the base shape.
• At this release, operations with a Python scalar operand (e.g., x + 1.0) and operations first compiled with the same tensor as both operands (e.g., x + x) cannot use the C++ warm cache either. Compiling first with two distinct tensors that have the same input profile (e.g., x + y with matching shapes and dtypes) lets later x + x calls reuse the C++ warm-cache entry.

Because the unit of compilation is one operator, an input shape that varies per call costs one program per operator per shape, not one per graph, and both cache levels live in the process that built them. RBLN PyTorch also sets torch._dynamo.config.specialize_float = True, so a Python float argument that changes between calls is a new program as well. Pad or bucket inputs to a fixed set of shapes before porting a model. For details, see Troubleshoot › Recompile limit exceeded.

Pre-dispatch checks¶

Before a compiled-kernel operator is sent to the device, a set of checks decides whether it must fall back instead. In order:

The dtype, scalar, and NaN/Inf checks are decided in the C++ shim and carry the why: reason shown above. The other checks run in the Python wrapper after the shim has missed, so a call that trips one of them misses the warm cache on every call: in an explain() report it appears as a repeating dispatch/recompile rather than in the dispatch/cpu_fallback counter, the fallback log names the operator, and each device-resident input is downloaded for the CPU kernel. The NaN/Inf scan reads the input on the host, so it too costs a download when the input's latest data is on the device; TORCH_RBLN_DEPLOY=ON skips the scan on the assumption that inputs are valid.

Each check can be skipped individually with TORCH_RBLN_DEV_DISABLE_OP_CPU_FALLBACK for development only; see Environment Variables › Development-only options.

Graph execution and input shapes¶

torch.compile(model, backend="rbln") compiles the whole captured graph as one program instead of one program per operator. The backend is part of the RBLN Compiler and is documented there: the option reference in Python API › torch.compile API and a model walkthrough in PyTorch ResNet50 › Using the torch.compile() API. This section covers what RBLN PyTorch adds around it: device-resident inputs and outputs, the fallback on failure, and the shape rules. The rbln backend is registered on the first torch.compile call, so no import is needed beyond import torch. Use dynamic=False, as the RBLN Compiler always compiles for fixed input shapes. With dynamic=True, code that reads tensor sizes (x.shape[0], view) fails on the second shape with Dynamic shape not supported for rbln backend. Under the default fallback settings that failure is handled by the compile_error fallback and the whole graph runs on the CPU.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import torch


def compute(x, y):
    return torch.relu(x + y)


x = torch.randn(128, 128, dtype=torch.float16, device="rbln:0")
y = torch.randn_like(x)
compiled_compute = torch.compile(compute, backend="rbln", dynamic=False)

with torch.inference_mode():
    first_result = compiled_compute(x, y)  # compiles for this input shape
    next_result = compiled_compute(x, y)  # same input profile: replays the program

    smaller_x = torch.randn(64, 128, dtype=x.dtype, device=x.device)
    smaller_y = torch.randn_like(smaller_x)
    smaller_result = compiled_compute(smaller_x, smaller_y)  # new shape: compiles again

The first two calls share an input profile and reuse the program. The third changes the shape from (128, 128) to (64, 128); with no matching cache entry the graph is compiled again. Measured on a Rebel100™ card for a two-operator step: the first call took about 800 ms, the replay 0.2 ms, and the new shape about 400 ms. With TORCH_RBLN_DISABLE_FALLBACK=compile_error, a compilation failure raises an error instead of running the whole function on the CPU. Set this environment variable when developing code that uses torch.compile. In a service whose input sizes vary, fix the batch size and sequence length and pad or bucket inputs to them so that shape changes at run time do not trigger compilation.

options is passed through to the RBLN Compiler: num_devices overrides the RSD group size, cache_dir chooses where artifacts are written, mode selects compile-only builds, and npu names the target SoC. torch._dynamo.reset() clears RBLN PyTorch's compile cache and drains in-flight work before device buffers are released.

Choose torch.compile when a model is made of many small operators, when an operator outside the eager tables is on the hot path, or when the program should be partitioned across an RSD group. Keep eager for development and debugging, and for glue code around a compiled region. Keeping the whole step on the device describes how the two combine in a serving step.

Views and strided copies¶

View operators (view, as_strided, permute, narrow, expand, select, squeeze, unsqueeze) are metadata-only on RBLN as on CUDA; they share storage with their base and move no data. The difference appears when a view is consumed:

• A compiled-kernel operator that receives a recognized view of a contiguous base replaces the view with its base and replays the view chain inside the compiled wrapper, so the operation stays on the device. Comparison operators (eq, ne, lt, le, gt, ge) are the exception: their view inputs are materialized with .contiguous() first.

• A contiguous view whose storage offset is not zero, such as base[1:, :], trips the storage-offset check and runs on the CPU:

  1 2 3 4 5 6 7 8

  import torch base = torch.randn(128, 128, dtype=torch.float16, device="rbln:0") part = base[1:, :] print(part.is_contiguous(), part.storage_offset()) # True 128 with torch.inference_mode(): result = part + part # CPU fallback: storage_offset != 0

  Creating the view and executing the addition are separate events; only the second one falls back.

• A device-to-device copy_ from a strided view runs on the device without a host round-trip.

  The runtime either issues a strided device copy directly or, when the view is a recognizable chain of single-step view operations from a contiguous base, replays that chain as a small compiled graph so the compiler lowers it on the device.

• A view chain the runtime cannot classify is materialized with .contiguous() through the host, once per process with a warning. explain() reports this as a host_bounce signal.

• When the output tensor specified by out= is non-contiguous, the runtime writes the result to a temporary contiguous buffer and then copies it to the output tensor. This adds one allocation and one copy. Specify a contiguous output tensor with out= in frequently executed code.

What falls back¶

Operators with their own device implementation follow the conditions in Supported operations. An operator that the CPU does not support either, or an invalid input, raises.

The compiled-function fallback converts the call arguments to CPU but not module parameters or tensors the function reads from outside its arguments. If those stay on rbln, the CPU execution raises a device-mismatch error. Under this fallback the whole graph runs on the host, so when you are checking whether a step stays on the device, disable the compile_error category to turn compilation failures into errors.

Fallback cost¶

A fallback moves every tensor argument to the host, runs the CPU kernel, and moves the outputs back to the device. For a lazily allocated tensor whose latest data still lives in the host view, the device-to-host step is free; for a materialized tensor it is a DMA, preceded by a synchronization. The >> physical d2h line of an explain() report tells the two cases apart, so read fallback counts together with the transfer volume. Fallbacks of operators without a device kernel, and of operators the Python dispatch layer hands to the CPU, are logged at INFO level with the operator name and, when traceable, the source location; fallbacks decided by the pre-dispatch checks are visible in explain() only. See Troubleshoot › CPU fallback logging.

Frequently hit fallback operators have host fast paths that bypass the generic boxed dispatcher; see Operators pinned to CPU. Operators without one are listed as optimization candidates in explain() output.

Fallback control¶

To surface an unsupported operator or a compilation failure as an error instead of running it on the CPU, list the category in TORCH_RBLN_DISABLE_FALLBACK before the process starts:

1

TORCH_RBLN_DISABLE_FALLBACK=compile_error,unsupported_op python application.py

Each category blocks exactly its own path: compile_error the compiled-function wrapper's fallback, unsupported_op the generic fallback for operators with no RBLN registration, strided_copy_error the fallback taken when the device's strided copy engine fails. The pre-dispatch checks (dtype, scalars, NaN/Inf, and the others in the table above) are not categories and are not affected, and operators registered explicitly to the CPU fallback path (Operators pinned to CPU) always run on the host by design. The category table, the default behaviors, and when each category is read are in Environment Variables › Execution and fallback.

NPU count per execution¶

How many NPUs an operator requests on ATOM™ follows these rules. Multi-NPU execution also requires that the compiler can partition the operator.

• Eager compute path: 1.
• Graph execution: the NPU count of the logical device that holds the input tensors by default. options={"num_devices": N} in torch.compile overrides it with N.

For example, with rbln:0 grouping four ATOM™ NPUs and the inputs on rbln:0:

Non-blocking copies¶

With non_blocking=True, the Python call can return while the transfer is still in progress. Tensor.to() and Tensor.copy_() take the asynchronous path only when all of the following hold:

• Source and destination have the same shape and dtype. A copy that needs a dtype conversion or a broadcast is synchronous.
• Both tensors are contiguous.
• For a host-device copy, the host buffer comes from the RBLN pinned allocator (pin_memory=True or Tensor.pin_memory()).
• For a host-to-device copy, the destination already resides in device memory as one flat region.

A freshly created lazy tensor fails the last condition: it has no device memory yet, so the copy lands in host memory and the upload happens when an NPU operation consumes the tensor. torch.rbln.bind_device_memory() allocates the flat region up front for a tensor that covers its whole storage (contiguous, storage offset zero); a non_blocking=True copy from a pinned host tensor into that buffer starts an asynchronous transfer. At this release, once the buffer has been used as an input to an NPU operation, eager or compiled, the runtime re-lays it out for that operation's input layout and later non_blocking=True copies into it become synchronous, so asynchronous upload is observable only before the buffer's first use; on the input buffer of a repeated step it becomes one synchronous host-to-device copy that completes inside the copy call. explain() counts synchronous uploads under host->device push and does not count transfers that took the asynchronous path.

A copy that does not meet the conditions is synchronous, and the runtime drains pending asynchronous transfers before switching paths; the switch itself raises no error. A copy between two RBLN logical devices is handed to the runtime's device-to-device copy, which may travel through host memory, and explain() does not count it.

While an asynchronous transfer is in flight: keep both buffers alive until it completes, do not modify the source or reuse the destination before it completes, and wait for completion before reading the destination on the CPU.

1
2
3
4
5
6
7
8

import torch

host = torch.empty(1 << 20, dtype=torch.float16, pin_memory=True)
dev = torch.empty(1 << 20, dtype=torch.float16, device="rbln")
torch.rbln.bind_device_memory(dev)  # give the destination its device memory up front

dev.copy_(host, non_blocking=True)  # returns before the transfer completes
torch.rbln.synchronize()  # wait for every pending transfer and stream on the current device

Streams¶

torch.rbln.Stream and torch.rbln.Event mirror torch.cuda.Stream and torch.cuda.Event, and the generic torch.Stream(device="rbln") and torch.Event(device="rbln") work as well. Work submitted to one stream runs in submission order. Completion is tracked per stream, so query() on one stream can return True while another still has work. To order work across streams, record an event and have the consuming stream wait on it; within one device that wait is a device-side fence and does not block the host.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import torch

x = torch.ones(1024, dtype=torch.float16, device="rbln")

s = torch.rbln.Stream()
with torch.rbln.stream(s):
    y = x * 2  # work issued while s is current
e = torch.rbln.Event()
e.record(s)
torch.rbln.current_stream().wait_event(e)

Events and waiting¶

An event marks a submission point on a stream. Work submitted to the same stream after the record is outside the event's scope.

torch.rbln.synchronize(device) waits for the logical device's pending asynchronous transfers and for the work on all of its streams; without an argument it uses the current device. Use an event to wait for a specific submission point and synchronize() to wait for the whole device.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import torch

with torch.rbln.device(0):
    device_buf = torch.empty(1 << 24, dtype=torch.float16, device="rbln")
    torch.rbln.bind_device_memory(device_buf)
    host_src = torch.ones_like(device_buf, device="cpu", pin_memory=True)
    done = torch.Event(device="rbln")

    device_buf.copy_(host_src, non_blocking=True)
    done.record()
    ready = done.query()
    print("query() right after record():", ready)
    if not ready:
        done.synchronize()  # the transfer is in flight: wait for it

    host_src.zero_()  # the transfer is complete: the source may be reused

query() right after record(): False

False right after the record means an asynchronous transfer was still in flight when the copy returned. True means the work was already complete; this value alone cannot distinguish a synchronous copy from an asynchronous one that finished quickly. If the copy call's return time grows with the transfer size, the copy is taking the synchronous path. The buffer rules above apply while you poll.

Measuring time¶

Without elapsed_time(), measure a region with a host timer. Warm up first so compilation and initialization are excluded, synchronize before starting the timer, and stop it only after the work you are measuring has completed. The following measures the time until the result is available on the CPU:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import time
import torch

x = torch.randn(128, 128, dtype=torch.float16, device="rbln:0")
y = torch.randn_like(x)

with torch.inference_mode():
    for _ in range(3):
        (x + y).cpu()  # warm-up: compile, then bring the result to the host

    torch.rbln.synchronize(0)
    start = time.perf_counter()
    result = (x + y).cpu()
    elapsed_seconds = time.perf_counter() - start

print(elapsed_seconds)

The measurement includes dispatch, execution, and the download of the result. If it is longer than expected, read the host->device push and physical d2h lines of a torch.rbln.explain() report for synchronous copies and host round-trips. Time spent inside the NPU is analyzed with the RBLN Profiler.

The three states of a tensor's data¶

The runtime tracks, per storage, whether host memory or NPU memory holds the latest data. This page names the states as follows:

Transfers happen only at state changes, so the goal restated is:

• Inputs go host-latest → in sync once, at the start of the step.
• Intermediates enter the next operator device-latest and no host code reads them.
• Only the results go device-latest → in sync, once, at the end of the step.

One line in the middle of a step that reads a device-latest tensor on the host inserts a D2H and a synchronous wait; a host-latest tensor created on every step repeats an H2D.

Code that crosses the host¶

The last row applies to eager execution only; the inputs and outputs of a compiled graph are not scanned. Its cost is visible directly: for a device-latest z consumed by one eager operator, z * y performs one D2H of the full size of z with TORCH_RBLN_DEPLOY unset and none with TORCH_RBLN_DEPLOY=ON.

The eager and graph boundary¶

A compiled graph is the most reliable way to keep a step on the device. At its boundary:

• A graph input that is device-latest or in sync costs no transfer. A host-latest input is pushed on every call.
• Passing a graph's output to the next graph as input costs no transfer.
• An eager operator that consumes a graph's output follows the NaN/Inf rule above.
• Inside the graph, dtype and shape constraints are handled by the compiler. An operator it cannot lower is split out as a host operator and runs inside the compiled artifact; RBLNCompiledModel.inspect() in the RBLN Compiler's Python API reports it under host_op_reason.
• An operator the frontend cannot convert fails compilation with NotImplementedError: The following operators are not implemented: [...]. Under the default settings the compile_error fallback then runs the whole graph on the CPU. When checking whether a step stays on the device, disable that fallback so the failure surfaces as an error (see Fallback control).

Build the body of a step as one compiled function and use eager operators only for preparation and wrap-up outside it.

Recommended step structure¶

1. Constants and weights move to RBLN once at program start. Do not create tensors inside the step.
2. The input buffer is created once with torch.empty and filled with copy_ on every step. This is the step's only H2D. With torch.rbln.bind_device_memory() the transfer happens at the copy; without it, at the first NPU operator that consumes the buffer. Either way it is one transfer per step. A non_blocking=True copy from a pinned host tensor is asynchronous only until the buffer's first use, so do not plan on overlap in a repeated step.
3. The step body is one function compiled with torch.compile(backend="rbln", dynamic=False). Fix input shapes by padding or bucketing, use one dtype (float16 or bfloat16), make the last dimension a multiple of 64, and pass offset-zero contiguous inputs.
4. No host reads inside the step. Express branches as device operations; collect logging and validation at the end.
5. Results come back once at the end with .cpu(). With several outputs, pack them into one tensor inside the graph and download once.
6. If the step must be eager, set TORCH_RBLN_DEPLOY=ON, meet the conditions of item 3 for every operator, and replace scalar operands with tensor constants.
7. Wrap the step in torch.rbln.explain() and confirm it is [clean] with one host->device push and one physical d2h.

Example: a step that crosses the host and a step that stays on the device¶

The same computation is written both ways and the two explain() reports are compared. The first step contains several patterns from the table above.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

import torch

dev = "rbln:0"
weight = torch.randn(128, 128, dtype=torch.float16, device=dev)
scale = torch.full((128, 128), 0.5, dtype=torch.float16, device=dev)


def step_eager(x_host):
    x = x_host.to(dev)  # H2D on every step
    h = torch.relu(x @ weight)
    if float(h.max()) > 1e4:  # value check: D2H and wait
        h = h * 0.5
    counts = torch.ones(
        128, 128, dtype=torch.int32, device=dev
    )  # constant created every step
    h = h * counts.to(torch.float16)  # dtype conversion through the host
    return (h * scale)[:, :100]  # last dimension not a multiple of 64


def step_body(x):
    return torch.relu(x @ weight) * scale


step_graph = torch.compile(step_body, backend="rbln", dynamic=False)

x_host = torch.randn(128, 128, dtype=torch.float16)

with torch.inference_mode():
    # Warm-up: run both paths once so compilation signals are excluded.
    step_eager(x_host).cpu()
    step_graph(x_host.to(dev)).cpu()

    with torch.rbln.explain(with_stack=True) as bad:
        result = step_eager(x_host).cpu()

    x_dev = torch.empty(128, 128, dtype=torch.float16, device=dev)
    torch.rbln.bind_device_memory(x_dev)
    x_dev.copy_(x_host)
    step_graph(x_dev).cpu()  # warm-up: attach the input buffer to the graph once

    with torch.rbln.explain(with_stack=True) as good:
        x_dev.copy_(x_host)  # step start: one H2D
        result = step_graph(x_dev).cpu()  # step end: one D2H

print(bad.report())
print(good.report())

The report of the step that crosses the host, abbreviated; timings, memory figures, and host-environment lines are omitted:

[overhead: 1 signal]  RBLN EXPLAIN   (region wall ... | device mem ... peak, reserved)

  host->device push (runtime): 3 pushes, ...

--------------------  -----  -----  -----------------------------------------------------------
Signal                Count  Bytes  Note
--------------------  -----  -----  -----------------------------------------------------------
host_bounce/d2d_copy      1    ...  real d2h DMA (costly) | try: make contiguous, or v2v engine
--------------------  -----  -----  -----------------------------------------------------------

  host_bounce/d2d_copy:
    at example.py:14(step_eager) <- ...

  >> physical d2h (real transfer): 3 copies, ... -- real device crossing

One step performs three H2D pushes and three real D2H transfers where the step below performs one of each; the input upload, the constant created inside the step, the dtype conversion, the value check, and the result download all contribute. The host_bounce/d2d_copy location points at the line that converts the int32 constant to float16.

The report of the step that stays on the device:

[clean]  RBLN EXPLAIN   (region wall ... | device mem ... peak, reserved)

  rebel runtime: ... in librbln
    v2h .../1  h2v .../1
  host->device push (runtime): 1 pushes, ...
  checked: host_bounce, v2v_slow, cpu_fallback, recompile -- none fired
  note: clean = no hidden host overhead, not "fast"

Only the input upload (h2v, one push) and the result download (v2h) remain, and no tracked signal fired. The call that first attaches the input buffer to the graph produces one extra transfer while the runtime re-lays the buffer out for the graph's input layout, which is why the example excludes that call as a warm-up. [clean] states that there is no hidden host round-trip, not that the step is fast; time inside the NPU is analyzed with the RBLN Profiler.