Troubleshoot¶

Each section covers one symptom or procedure: the message you see, why it happens, and the steps that resolve it. Start with the environment report whenever the symptom is an import or library error; it does not need an NPU.

1

python -m torch_rbln.diagnose

The module does not load the torch native libraries or the RBLN extension, so the report prints even when import torch itself fails.

The report's blocks and the values to check are described in Installation › Verification.

Missing librbln.so / librbln_runtime.so¶

The RBLN runtime libraries installed with rebel-compiler are not visible to the dynamic loader in the current Python environment.

To resolve the issue:

1. Activate the environment where rebel-compiler is installed. Confirm with pip show rebel-compiler or uv pip list | grep rebel.
2. Check whether LD_LIBRARY_PATH or PYTHONPATH from another stack changes the library search order. The diagnose report lists every candidate path in order; the loaded one is marked. Prefer a clean shell or only your project virtual environment.
3. Reinstall rebel-compiler and torch-rbln according to Installation if the setup is unclear.

ABI mismatch between torch-rbln and librbln.so¶

The ABI (Application Binary Interface) defines the binary interface between torch-rbln and librbln.so.

import torch loads torch-rbln through the autoload entry point, which checks that the librbln.so it loads implements the interface the wheel was built against. rebel-compiler exports two numbers, the interface it implements and the oldest consumer contract it still accepts; torch-rbln records the interface it compiled against and requires that it falls inside that window. Outside the window the import fails with an RBLN ABI mismatch message naming the library in use and both versions, instead of continuing into an undefined symbol crash or corruption inside the runtime.

To resolve the issue:

1. Install torch-rbln and rebel-compiler from the same SDK release. The release notes list the versions that ship together; Installation › Version pairing has the commands.
2. Run python -m torch_rbln.diagnose to see which librbln.so was loaded, the interface the wheel was built against, the range the runtime accepts, and the resulting verdict.

A runtime that predates this check, or a torch-rbln built against one, leaves nothing to compare, so the import warns and continues instead of failing.

If a matching wheel is not available yet, TORCH_RBLN_SKIP_ABI_CHECK skips the check so the mismatched pair loads together:

1

export TORCH_RBLN_SKIP_ABI_CHECK=1   # accepts 1 / ON / TRUE / YES

Skipping the check hides the incompatibility rather than removing it; see Environment Variables › ABI check before relying on it.

Import crash with undefined symbol¶

import torch aborts with ImportError: ... undefined symbol: ... naming a rbln_ or c10::rbln symbol.

This is the failure the ABI check exists to prevent. It appears when the check was skipped with TORCH_RBLN_SKIP_ABI_CHECK, when the runtime predates the check, or when a second librbln.so from another installation was mapped first through LD_LIBRARY_PATH.

1. Unset TORCH_RBLN_SKIP_ABI_CHECK and LD_LIBRARY_PATH, then run python -m torch_rbln.diagnose and check which librbln.so is marked as loaded.
2. Install torch-rbln and rebel-compiler from the same release.

No visible NPU¶

torch.rbln.device_count() returns 0, torch.rbln.is_available() returns False, or a device allocation fails with a runtime error that mentions device enumeration.

Both query functions never raise; they return 0 and False when the runtime is absent, no NPU is visible, or the device-mapping configuration is malformed. The malformed case also prints a warning, and the full error is raised by torch.rbln.current_device(), torch.rbln.set_device(), or the first allocation.

1. Confirm the driver sees the cards: rbln-smi lists every NPU on the host. If the table is empty, follow the RBLN Driver installation guide.
2. Check RBLN_VISIBLE_DEVICES (RBLN_DEVICES is a deprecated alias). An id outside the host's range leaves the process with no visible device.
3. Check RBLN_NPUS_PER_DEVICE and RBLN_DEVICE_MAP. A group size other than 1, 2, 4, 8, 16, 32, a duplicate id, or an id outside the visible pool is a configuration error. Print the plan with torch.rbln.device_summary().
4. In a container, confirm the device nodes are passed through with the RBLN Container Toolkit.

Physical NPU in use¶

The first device use raises RuntimeError with a message of this form:

rbln_register_device_id failed for rbln:0 on physical NPU(s) [0] (rc=<N>); the device(s) may be in use by another process or hold stale allocations. Free the device(s) or adjust RBLN_DEVICES.

The runtime could not claim one of the NPUs the device mapping asked for. On a shared host this is another process holding the card; after a crash it can be a process that has not released its allocations. When several logical devices were being registered, the message ends with a note naming the ones this process already claimed; they stay claimed until the process exits.

1. Find the holder with rbln-smi, whose process table lists the PID and allocated memory per NPU, and wait for it or stop it.
2. Point this process at free NPUs with RBLN_VISIBLE_DEVICES, or reduce the group size in RBLN_NPUS_PER_DEVICE or RBLN_DEVICE_MAP.
3. If no process holds the card and the error persists, report it with the rbln-smi output (Issue reporting).

Device out of memory¶

An allocation, or the first device operation on a tensor, raises RuntimeError with one of these messages:

rbln_malloc_lazy failed (rbln:0, <BYTES> bytes); the device may be out of memory or hold stale allocations
rbln_malloc_eager failed (rbln:0, <BYTES> bytes); the device may be out of memory or hold stale allocations

The device memory of that logical device is exhausted. Because allocations are lazy, the failure often surfaces at the first device operation that materializes a tensor rather than at the torch.empty call that reserved it, so the line that raises may not be the line that allocated.

1. Read torch.rbln.memory_reserved() and torch.rbln.memory_allocated() before the failing step. A large gap is cached memory that torch.rbln.empty_cache() returns to the runtime.
2. Compiled programs hold device buffers of their own, and shape churn multiplies them; torch._dynamo.reset() releases them. See Recompile limit exceeded.
3. Use torch.inference_mode() during inference to avoid unnecessary memory overhead.
4. If another process shares the card, its allocations count too; the Memory(used/total) column of rbln-smi shows the total.

Core dump collection¶

If the process crashes while running RBLN PyTorch, send the generated core dump file to client_support@rebellions.ai together with the information listed under Issue reporting.

1. Remove the ulimit restriction:

   $ ulimit -c unlimited
   

2. Verify that the restriction has been removed:

   $ ulimit -c
   unlimited
   

3. Re-run the affected script. When the crash occurs, a core dump file is created under /var/crash:

   $ ls /var/crash
   -rw-r----- 1 rebel1    root   779026 Jul  2 17:50 /var/crash/_usr_bin_python3.10.2029.crash
   -rw-r----- 1 rebel2    root 94849351 Jun 25 18:27 /var/crash/_usr_bin_python3.10.2035.crash
   

CPU fallback logging¶

When RBLN PyTorch does not support a PyTorch operator or a specific data type, that operation runs on the CPU so execution can continue. This improves model compatibility, but those operations do not benefit from NPU acceleration, so identifying them is useful during optimization.

By default, the RBLN PyTorch log level is WARNING, so CPU fallback messages are not displayed. CPU fallback operations are logged at the INFO level. To identify operators that run on the CPU, set TORCH_RBLN_LOG_LEVEL to INFO; every level is described in Environment Variables › Logging.

$ export TORCH_RBLN_LOG_LEVEL=INFO

To restore the default setting:

$ export TORCH_RBLN_LOG_LEVEL=WARNING

With this setting, running a model in eager mode prints a log line when an operator without a device kernel, or an operator that the Python dispatch layer hands to the CPU, runs on the CPU instead of an RBLN NPU. The line includes the operator name and, when traceable, the source code location. Fallbacks decided by the pre-dispatch checks (dtype, all-scalar inputs, NaN/Inf) do not produce a log line; torch.rbln.explain() reports them with a why: reason.

[2026-03-27 00:00:00.000][I] `aten::pow` op ran on CPU instead of RBLN
/transformers/models/llama/modeling_llama.py:72: UserWarning: TRACE
  variance = hidden_states.pow(2).mean(-1, keepdim=True)

Common causes, in order of frequency:

For a structured view instead of a log stream, wrap the region with torch.rbln.explain(); its dispatch/cpu_fallback block lists each operator that a pre-dispatch check sent to the CPU, with a count, a why: reason, and, with with_stack=True, the call site. Operators that have no device kernel or are pinned to the CPU are not counted there and appear only in the log. See the tutorial. To turn an unregistered operator into an error, set TORCH_RBLN_DISABLE_FALLBACK=unsupported_op before launching the process; the dtype, scalar, and NaN/Inf fallbacks are pre-dispatch checks that this variable does not cover. See Environment Variables.

Recompile limit exceeded¶

A warning reports that the Dynamo cache limit was reached, a region that was fast becomes slow, or the process raises torch._dynamo.exc.FailOnRecompileLimitHit.

Eager execution compiles each operator per shape and dtype through Dynamo. RBLN PyTorch raises the per-frame cache limit to 64 and configures Dynamo to fail hard when the limit is exceeded, so shape churn is detected instead of recompiling silently forever. When the limit is hit, the compile wrapper resets Dynamo and retries the call once; if the limit is hit again, the wrapper falls back to CPU execution for that call, or raises when TORCH_RBLN_DISABLE_FALLBACK contains compile_error. TORCH_LOGS=recompiles prints the reason for each recompile, as on any Dynamo backend. RBLN PyTorch also sets torch._dynamo.config.specialize_float = True, so a Python float argument that changes between calls is a new program.

1. Find the source of shape churn with TORCH_LOGS=recompiles. Common sources are growing sequence lengths without padding, data-dependent shapes, and Python floats that change on every call.
2. Pad or bucket inputs to a fixed set of shapes.
3. If the shape set is legitimately larger than 64, raise the limit before the first device operation:

1
2
3

import torch

torch._dynamo.config.cache_size_limit = 256

Slow first call¶

The first execution of an operator, a model, or a torch.compile graph takes seconds to minutes; later calls are fast.

This is compilation, not a hang. Each new operator shape and dtype in eager mode, and each new graph under torch.compile, is compiled by the RBLN Compiler on first use and cached. To confirm, wrap the first and a later call with torch.rbln.explain_steady(); the dispatch/recompile signal disappears in the steady-state region. To keep the first call short, fix input shapes so that the compiled programs are reused, and keep the process alive between requests. Execution Semantics › Graph execution and input shapes gives measured compile, replay, and new-shape timings.

Memory statistics¶

torch.rbln.memory_allocated() and torch.rbln.memory_stats() report lower values than expected right after creating tensors on rbln, or report zero after allocating large tensors.

This is expected: device allocations are lazy, so a tensor counts only after a device operation materializes it, and memory APIs report device memory only. Compiled programs cached by Dynamo also hold device memory. See Execution Semantics › Memory statistics and the caching allocator for the full explanation and the recommended measurement pattern.

Results differ from a CPU run¶

Outputs on rbln differ from the same script on CPU, or greedy decoding picks a different token.

Element-by-element equality with a CPU float16 run is not expected. Device 16-bit arithmetic runs in the runtime's own compute format, which carries fewer mantissa bits than IEEE float16; see Execution Semantics › Precision and dtypes.

1. Recompute the reference on CPU in float32 and compare with a tolerance (torch.testing.assert_close), not with equality.
2. Leave TORCH_RBLN_DEPLOY unset while debugging so an input containing NaN or Inf still routes to CPU and appears in the fallback log.
3. Run with TORCH_RBLN_LOG_LEVEL=INFO. An operator that fell back computed on the host, so a difference that disappears when the operator falls back is a device-precision difference, not a wrong result.
4. If the difference is large or grows with sequence length, report it with both outputs and the dtype the model ran in (Issue reporting).

Issue reporting¶

Report bugs and unexpected behavior through one of the channels below. Include the items in the checklist so the report can be reproduced without a follow-up.

Checklist:

1. The environment report: python -m torch_rbln.diagnose output, which includes the torch-rbln and rebel-compiler versions and the ABI verdict.
2. The driver version from rbln-smi and the SDK release from the release notes you installed from.
3. A minimal script that reproduces the problem, with fixed input shapes and dtypes, and the exact command line including environment variables (TORCH_RBLN_*, RBLN_*).
4. The full log with TORCH_RBLN_LOG_LEVEL=INFO, and the torch.rbln.explain() report if the problem is performance.
5. For a crash, the core dump collected as described in Core dump collection.
6. For a wrong result, a reference output computed on CPU in float32, the observed output, and the dtype the model ran in.

See also¶

• Installation › Verification - the checks to run after installing or upgrading
• Environment Variables - every variable referenced above
• Execution Semantics - dtype policy, lazy memory, and fallback rules behind these symptoms