Skip to content

Python API

RBLN compile API

RBLNCompiledModel

A class that holds the compiled binaries.

An instance is returned by the compile_from_* functions, or loaded from a saved .rbln file by passing its path to the constructor.

__init__(compiled_model)

Load a compiled model from a .rbln file.

Parameters:

Name Type Description Default
compiled_model PathLike or str

Path to the .rbln file.

 required

Raises:

Type Description
TypeError

If compiled_model is not a path.

save(path)

Serialize and save the model to the disk, .rbln formatted file.

Parameters:

Name Type Description Default
path PathLike

Path to save serialized data

 required

create_runtime(*, device=0, tensor_type='np')

Create a runtime from this compiled model. Note that this function is exclusive to create_async_runtime. Once you create a runtime by create_runtime with the instance, you can't call create_async_runtime.

Parameters:

Name Type Description Default
device int

The device ID of the NPU to use for execution. Defaults to 0.

 0
tensor_type str

The object type of the tensor used in the run function. Possible values are:

  • "np": Uses np.ndarray type.
  • "pt": Uses torch.Tensor type.

Defaults to "np".

 'np'

Returns:

Name Type Description
Runtime Runtime

Runtime object that can be run on the RBLN NPU.

Raises:

Type Description
RBLNRuntimeError

If the requested NPU is missing or does not match the compiled model.

create_async_runtime(*, device=0, tensor_type='np', parallel=None)

Create an asynchronous runtime from this compiled model. Note that this function is exclusive to create_runtime. Once you create an asynchronous runtime by create_async_runtime with the instance, you can't call create_runtime.

Parameters:

Name Type Description Default
device int

The device ID of the NPU to use for execution. Defaults to 0.

 0
tensor_type str

The object type of the tensor used in the run function. Possible values are:

  • "np": Uses np.ndarray type.
  • "pt": Uses torch.Tensor type.

Defaults to "np".

 'np'
parallel int

The number of threads used to prepare and queue input data for the NPU's input buffer. The NPU supports double buffering, allowing it to process one input while the next is being prepared. Possible values are:

  • 1: Uses a single thread to prepare inputs (default).
  • 2: Uses two threads to prepare inputs, enabling double buffering. This can potentially improve performance when input preparation is time-consuming, as one thread can prepare the next input while the NPU is still processing the current one.

This parameter helps manage the NPU's input buffer more efficiently, ensuring that the next input is ready as soon as the current computation finishes. Only applicable to AsyncRuntime due to its non-blocking nature.

Note: Use parallel = 2 with caution, as it may not work on some models (e.g. LLM). It's recommended to benchmark your specific use case to determine if it provides a performance benefit.

 None

Returns:

Name Type Description
AsyncRuntime AsyncRuntime

Asynchronous runtime object that can be run on the RBLN NPU.

Raises:

Type Description
ValueError

If parallel is not 1 or 2.

get_total_device_alloc(parallel=1)

Retrieve the total device memory allocation (in bytes) required for the compiled graph across all NPUs.

Parameters:

Name Type Description Default
parallel int

If parallel > 1, the returned allocation size accounts for additional buffer requirements when operating in non-blocking (asynchronous) mode. Defaults to 1.

 1

Returns:

Name Type Description
int int

The total device memory allocation (in bytes) required for the compiled graph.

get_alloc_per_node(parallel=1)

Retrieve the device memory allocation (in bytes) required for the compiled graph on each individual NPU.

Parameters:

Name Type Description Default
parallel int

If parallel > 1, the returned allocation size accounts for additional buffer requirements when operating in non-blocking (asynchronous) mode. Defaults to 1.

 1

Returns:

Type Description
List[int]

List[int]: A list containing the device memory allocation (in bytes) for each NPU. The length of the list equals the num_devices the model was compiled for.

inspect(path) classmethod

Return metadata about the compiled model without loading it into host memory.

This method retrieves essential details such as:

  • Estimated memory usage per device
  • Compiler version used for compilation
  • I/O Tensor's name, shape, and dtype
  • Name of the compiled model
  • Target NPU type
  • Number of required devices (num_devices)
  • Unique identifier (UUID) for the compiled model
  • Reasons for graph breaks (If device_func_count == 1, it means no graph breaks occurred)

Parameters:

Name Type Description Default
path PathLike

Path to the compiled RBLN model file.

 required

Returns:

Type Description
Dict[str, Any]

Dict[str, Any]: Dictionary containing metadata information about the compiled model.
Example 
1
2
3
4
from rebel import RBLNCompiledModel

metadata = RBLNCompiledModel.inspect("/path/to/compiled/model.rbln")
print(metadata)
Output 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
{
    'alloc_per_node': [8002732032, 7969177600, 7969177600, 7969177600],
    'compiler_version': '0.7.2',
    'subgraph': {
        'stats': [
        {
            'compile_id': 0,
            'device_func_count': 2,
            'host_op_reason': {
            'rbln_rebel_descriptor_main_0': {
                'aten::clamp_min_0': 'BreakReason.WrongShape',
                'aten::lt_0': 'BreakReason.Unsupported',
                'aten::sum_0': 'BreakReason.WrongShape',
                'aten::to_0': 'BreakReason.WrongShape',
                'aten::to_1': 'BreakReason.WrongShape'
            },
            'rbln_rebel_descriptor_main_6': {
                'aten::clamp_min_0': 'BreakReason.WrongShape',
                'aten::cumsum_0': 'BreakReason.Unsupported',
                'aten::lt_0': 'BreakReason.Unsupported',
                'aten::lt_1': 'BreakReason.Unsupported',
                'aten::sum_0': 'BreakReason.WrongShape',
                'aten::to_0': 'BreakReason.WrongShape',
                'aten::to_1': 'BreakReason.WrongShape',
                'aten::to_2': 'BreakReason.WrongShape',
                'aten::transpose_2': 'BreakReason.WrongShape'
            }
            }
        }
        ]
    },
    'inputs': [
        {
            'dtype': 'float32',
            'name': 'inputs_embeds',
            'shape': [1, 128, 4096]
        },
        {
            'dtype': 'float32',
            'name': 'attention_mask',
            'shape': [1, 1, 128, 32768]
        },
    ],
    'name': '3c74b1',
    'npu': 'RBLN-CA25',
    'outputs': [
        {
            'dtype': 'float32',
            'shape': [1, 1, 32064]
        }
    ],
    'num_devices': 4,
    'uuid': '5b8183b8e75e11efa2ebece7a7056f04'
}

compile_from_torch(mod, input_info=None, example_inputs=None, *, npu=None, num_devices=None, model_trace_method=None)

Compile a model from torch.nn.Module.

Parameters:

Name Type Description Default
mod Module

A PyTorch model (torch.nn.Module) or a callable.

 required
input_info List[Tuple[str, List[int], DType]]

Input configuration information. Supports two formats:

Single Input Configuration: A list of input information entries, each described as a (name, shape, dtype) tuple.

  • name : str - Name of the input tensor
  • shape : List[int] - Shape of the input tensor
  • dtype : str or torch.dtype (i.e., "float32" or torch.float32)

Bucketing (Multiple Input Configurations): A list of input configurations, where each configuration is itself a list of input information. This enables the compiled model to support multiple input shapes efficiently within a single runtime, allowing automatic bucket selection based on input shape at inference time.

For detailed bucketing examples and best practices, see bucketing tutorial.

 None
example_inputs List[Tensor]

A list of example input torch tensors that can be used for tracing. If None, tracing may use default dummy inputs obtained from input_info.

 None
npu str

The identifier of the target NPU for compilation. If None, the function will use the NPU installed on the host machine. If an NPU is not installed on the host machine, an error will be raised. Defaults to None.

 None
num_devices int

The number of NPU devices to distribute the model across. The parallelization strategy (tensor / pipeline / etc.) is decided by the compiler. If None, the model is compiled for a single device. Defaults to None. (tensor_parallel_size is a deprecated alias for this argument.)

 None
model_trace_method str

The method used to trace the given model. If unspecified, the function will first try to trace the model with torch.export.export non-strict mode. If it fails, it will try to trace the model with torch.jit.trace. If 'export' is given, the model is traced by torch.export.export non-strict mode. If 'export_strict' is given, the model is traced by torch.export.export strict mode. If 'jittrace' is given, the model is traced by torch.jit.trace.

 None

Returns:

Name Type Description
RBLNCompiledModel RBLNCompiledModel

Compiled model that can be run on the RBLN NPU.

compile_from_torchscript(mod, input_names=None, *, npu=None, num_devices=None)

Compile a model from torch.jit.ScriptModule, a result of torch.jit.trace function.

Note that the input shape and dtype information must be kept to compile a TorchScript model. When a TorchScript model is loaded with torch.jit.load, this information is not restored by default. Pass _restore_shapes=True to restore it.

1
2
mod = torch.jit.load("model.pt", _restore_shapes=True)
compile_from_torchscript(mod)

Parameters:

Name Type Description Default
mod TorchScript

A PyTorch jit-traced model.

 required
input_names Optional[List[Optional[str]]]

A list of input names. If input_names is specified as None, it'll derive name from mod, name of corresponding input in forward function.

 None
npu str

The identifier of the target NPU for compilation. If None, the function will use the NPU installed on the host machine. If an NPU is not installed on the host machine, an error will be raised. Defaults to None.

 None
num_devices int

The number of NPU devices to distribute the model across. The parallelization strategy (tensor / pipeline / etc.) is decided by the compiler. If None, the model is compiled for a single device. Defaults to None. (tensor_parallel_size is a deprecated alias for this argument.)

 None

Returns:

Name Type Description
RBLNCompiledModel RBLNCompiledModel

Compiled model that can be run on the RBLN NPU.

compile_from_tf_function(func, input_info, outputs=None, layout='NHWC', *, npu=None)

Compile a model from tf.function. Note that the input function should not be concretized by get_concrete_function method.

Parameters:

Name Type Description Default
func GenericFunction

A TensorFlow function

 required
input_info List[Tuple[str, List[int], DType]]

A list of input information entries, each described as a (name, shape, dtype) tuple. If the dtype is specified as None, it'll derive the dtype from dtype parameter.

 required
outputs Optional[Union[str, List[str]]]

A string or list of the name of output node(s) (Optional). If not specified, then the last node is assumed to be the graph output. This may be useful when the graph has multiple outputs.

 None
layout str

Layout of the tensor used internally in the model. One of "NHWC" or "NCHW"

 'NHWC'
npu str

The identifier of the target NPU for compilation. If None, the function will use the NPU installed on the host machine. If an NPU is not installed on the host machine, an error will be raised. Defaults to None.

 None

Returns:

Name Type Description
RBLNCompiledModel RBLNCompiledModel

Compiled model that can be run on the RBLN NPU.

compile_from_tf_graph_def(graph_def, outputs=None, layout='NHWC', *, npu=None)

Compile a model from TensorFlow GraphDef. This function allows you to compile TensorFlow V1.x legacy models. If you are using TensorFlow V2 as a default, we recommend compiling the model using compile_from_tf_function in its function form.

Parameters:

Name Type Description Default
graph_def GraphDef

A TensorFlow graph definition in the form of a protocol buffer

 required
outputs Optional[Union[str, List[str]]]

A string or list of the name of output node(s) (Optional). If not specified, then the last node is assumed to be the graph output. This may be useful when the graph has multiple outputs.

 None
layout str

Layout of the tensor used internally in the model. One of "NHWC" or "NCHW"

 'NHWC'
npu str

The identifier of the target NPU for compilation. If None, the function will use the NPU installed on the host machine. If an NPU is not installed on the host machine, an error will be raised. Defaults to None.

 None

Returns:

Name Type Description
RBLNCompiledModel RBLNCompiledModel

Compiled model that can be run on the RBLN NPU.

torch.compile API

compile(model=None, *, dynamic=None, backend='inductor', options=None)

Optimize the given model/function using TorchDynamo with the RBLN backend for execution on RBLN NPUs.

This function compiles the input model to run efficiently on RBLN NPUs. It leverages TorchDynamo for tracing and the RBLN backend for generating optimized code tailored to RBLN NPUs. To use this backend, run import rebel before calling torch.compile.

Parameters:

Name Type Description Default
model Callable

The module or function to optimize.

 None
dynamic bool or None

Use dynamic shape tracing. The RBLN backend does not support general dynamic shapes. Set this to False. A dimension marked with torch._dynamo.mark_dynamic is traced at its current size, and its range is passed to the compiler. This is an experimental feature and is supported only in limited cases.

 None
backend str or Callable

The backend to use.

  • To use the RBLN backend, set this to "rbln" and ensure import rebel is executed before calling torch.compile.
 'inductor'
options dict

A dictionary of options to pass to the backend. Notable options for the RBLN backend are:

  • cache_dir which specifies the directory where compiled artifacts should be stored.

  • mega_cache_only which stores compiled artifacts only in the session-wide mega-cache bundle instead of per-graph .rbln files in cache_dir. Intended for integrations that save and restore the bundle themselves, such as RBLN vLLM. Set cache_dir as well, because the bundle stages large artifacts there. Defaults to False.

  • mode which specifies the mode(s) of compilation. Accepts a single string or a list of strings.

    • "strict": raise an exception on compilation failure instead of falling back to eager mode.
    • "compile_only": compile and save the artifact only. No runtime is created, so the host needs no NPU. Calling the compiled model returns empty tensors with the output shapes and dtypes, and no computation runs. Requires cache_dir and npu. Can also be forced via RBLN_COMPILE_ONLY=1 (env wins over mode).

  • npu which specifies the target NPU for compilation. If None, the function will use the NPU installed on the host machine. If no NPU is installed on the host machine, an error will be raised. Defaults to None.

  • device which specifies the device ID of the NPU to use for execution. If None, the compiled graph runs on the device of its rbln input tensors (rbln:N runs on device N). If no input is an rbln tensor, device 0 is used (devices 0 to num_devices - 1 when num_devices is greater than 1). Defaults to None.

  • num_devices which specifies the number of NPUs to distribute the model across. The compiler decides the parallelization strategy. Defaults to None (single device). tensor_parallel_size is a deprecated alias for this option.

  • dtype which specifies the dtype the model runs in on the device. One of "float16", "bfloat16", torch.float16, or torch.bfloat16. Floating-point weights are converted to it at load, and every device tensor uses it. Quantized weights keep their own dtype, and only the activations use it. Host-side input and output dtypes are unchanged. Defaults to None (the compiler default).

  • model_trace_method which selects the tracing method passed through to compile_from_torch. One of "export" / "export_strict" / "jittrace" / None. If None (default), the backend first tries torch.export.export in non-strict mode and falls back to torch.jit.trace on failure. "export" forces non-strict export, "export_strict" forces strict export, and "jittrace" forces torch.jit.trace.

  • use_static_output which reuses a fixed device output buffer per output across runs instead of allocating a fresh one each call. Defaults to False.

 None
Example 
1
2
3
4
compiled_model = torch.compile(model,
                               backend="rbln",  # Specify the RBLN backend
                               options={"cache_dir": "./rbln_cache_dir"},
                               dynamic=False)

runtime API

RuntimeBase

A base class for runtime, providing common functionalities for both synchronous and asynchronous runtime.

The RuntimeBase class serves as a foundational class for the Runtime and AsyncRuntime classes. It encapsulates shared methods and attributes that manage the execution of the model.

num_threads property writable

The current number of threads used by the system.

Returns:

Name Type Description
int int

The current number of threads.

model_description()

Return a description of the model currently loaded in the runtime.

This method provides a summary of the model's architecture, including details about its inputs, outputs, and memory usage on the RBLN NPU.

Returns:

Name Type Description
str str

A string containing the model's description.

flush_reports()

Fetch and discard all pending reports from the runtime.

This method streams all available reports using _stream_reports() and then clears the internal _reports list.

get_reports()

Retrieve all pending reports from the runtime.

Returns:

Type Description
List[Dict[str, Any]]

List[str]: A list of report dictionaries in string format.

This method fetches all available reports and returns their contents while preserving their order.

Note

Timer reports are only available when the environment variable RBLN_RUNTIME_TIMER=1 is set. Without this setting, timer reports will not be generated.

Example 
1
2
3
4
5
# Before the runtime timer reports, the runtime must be executed.
result = runtime.run(input)

# One dictionary report is included per runtime execution.
print(runtime.get_reports())
Output 
1
2
3
[{'type': 'timer', 'default_fused_nn_pad_1': 1, 'rbln_rebel_descriptor_main_2': 922,
'default_fused_transpose': 34, 'default_fused_nn_pad': 1358, 'total': 2315,
'total_host': 1393, 'total_device': 922}]

get_elapsed_times()

Calculate the average total execution time of operations in microseconds.

Returns:

Name Type Description
float float

Average execution time in microseconds, or 0.0 if no timer reports are available.

This method iterates over the available reports, sums the total values for reports, and computes the average across all such reports.

Note

Timer reports are only available when the environment variable RBLN_RUNTIME_TIMER=1 is set. Without this setting, timer reports will not be generated.

get_elapsed_device_times()

Calculate the average device-side execution time of operations in microseconds.

Returns:

Name Type Description
float float

Average device execution time in microseconds, or 0.0 if no timer reports are available.

This method iterates over the available reports, sums the total_device values for reports, and computes the average across all such reports.

Note

Timer reports are only available when the environment variable RBLN_RUNTIME_TIMER=1 is set. Without this setting, timer reports will not be generated.

Runtime

A Runtime object for executing a compiled neural network on an NPU.

__init__(compiled_model, *, device=0, tensor_type='np', activate_profiler=False, timeout=None)

Initialize a Runtime object for executing a compiled neural network on an NPU.

Parameters:

Name Type Description Default
compiled_model str, PathLike, or RBLNCompiledModel

The path to the compiled RBLN model file (*.rbln) or an instance of RBLNCompiledModel.

 required
device int

The device ID of the NPU to use for execution. Defaults to 0.

 0
tensor_type str

The object type of the tensor used in the run function. Possible values are:

  • "np": Uses np.ndarray type.
  • "pt": Uses torch.Tensor type.

Defaults to "np".

 'np'
activate_profiler bool

Whether to activate profiling for this runtime instance. If set to True, profiling information (e.g., execution times) will be collected during execution. Useful for debugging and performance tuning. Defaults to False.

 False
timeout int

The timeout for the runtime in seconds. If the runtime exceeds this timeout, it will be terminated. If not provided, the runtime will use the default timeout value (60 seconds).

 None

Raises:

Type Description
TypeError

If compiled_model is neither a path nor an RBLNCompiledModel.

run(*input_args, out=None, **input_kwargs)

Run the compiled neural network with the given input tensors.

Parameters:

Name Type Description Default
*input_args Union[ndarray, Tensor]

Variable length argument list of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 ()
out Optional[List[Union[ndarray, Tensor]]]

An optional list or tensor to store the output tensors. If provided, it must contain pre-allocated tensors with shapes matching the network's output shapes. If not provided or set to None, new tensors will be allocated to store the outputs.

 None
**input_kwargs Union[ndarray, Tensor]

Arbitrary keyword arguments of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 {}

Returns:

Type Description
Union[ndarray, Tensor, List[Union[ndarray, Tensor]]]

The output tensor(s) of the neural network. The return depends on the network's architecture and can be either a single tensor or a list of tensors. The tensor type (numpy.ndarray or torch.Tensor) is determined by the tensor_type provided during the Runtime object's initialization.

forward(*input_args, out=None, **input_kwargs)

An alias for the run method.

This method is provided for compatibility with PyTorch's naming convention.

Parameters:

Name Type Description Default
*input_args Union[ndarray, Tensor]

Variable length argument list of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 ()
out Optional[List[Union[ndarray, Tensor]]]

An optional list or tensor to store the output tensors. If provided, it must contain pre-allocated tensors with shapes matching the network's output shapes. If not provided or set to None, new tensors will be allocated to store the outputs.

 None
**input_kwargs Union[ndarray, Tensor]

Arbitrary keyword arguments of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 {}

Returns:

Type Description
Union[ndarray, Tensor, List[Union[ndarray, Tensor]]]

The output tensor(s) of the neural network, as returned by the run method.

__call__(*input_args, out=None, **input_kwargs)

Allow the Runtime object to be called as a function.

This method is provided for convenience and compatibility with common neural network frameworks.

Parameters:

Name Type Description Default
*input_args Union[ndarray, Tensor]

Variable length argument list of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 ()
out Optional[List[Union[ndarray, Tensor]]]

An optional list or tensor to store the output tensors. If provided, it must contain pre-allocated tensors with shapes matching the network's output shapes. If not provided or set to None, new tensors will be allocated to store the outputs.

 None
**input_kwargs Union[ndarray, Tensor]

Arbitrary keyword arguments of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 {}

Returns:

Type Description
Union[ndarray, Tensor, List[Union[ndarray, Tensor]]]

The output tensor(s) of the neural network, as returned by the run method.

AsyncRuntime

An AsyncRuntime object for executing a compiled neural network asynchronously on an NPU.

__init__(compiled_model, *, device=0, tensor_type='np', parallel=None, timeout=None)

Initialize an AsyncRuntime object for executing a compiled neural network asynchronously on an NPU.

Parameters:

Name Type Description Default
compiled_model str, PathLike, or RBLNCompiledModel

The path to the compiled RBLN model file (*.rbln) or an instance of RBLNCompiledModel.

 required
device int

The device ID of the NPU to use for execution. Defaults to 0.

 0
tensor_type str

The object type of the tensor used in the run function. Possible values are:

  • "np": Uses np.ndarray type.
  • "pt": Uses torch.Tensor type.

Defaults to "np".

 'np'
parallel int

The number of threads used to prepare and queue input data for the NPU's input buffer. The NPU supports double buffering, allowing it to process one input while the next is being prepared. Possible values are:

  • 1: Uses a single thread to prepare inputs (default).
  • 2: Uses two threads to prepare inputs, enabling double buffering. This can potentially improve performance when input preparation is time-consuming, as one thread can prepare the next input while the NPU is still processing the current one.

This parameter helps manage the NPU's input buffer more efficiently, ensuring that the next input is ready as soon as the current computation finishes. Only applicable to AsyncRuntime due to its non-blocking nature.

Note: Use parallel = 2 with caution, as it may not work on some models (e.g. LLM). It's recommended to benchmark your specific use case to determine if it provides a performance benefit.

 None
timeout int

The timeout for the runtime in seconds. If the runtime exceeds this timeout, it will be terminated. If not provided, the runtime will use the default timeout value (60 seconds).

 None

Raises:

Type Description
ValueError

If parallel is not 1 or 2.

run(*input_args, **input_kwargs)

Run the compiled neural network asynchronously with the given input tensors.

Parameters:

Name Type Description Default
*input_args Union[ndarray, Tensor]

Variable length argument list of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 ()
**input_kwargs Union[ndarray, Tensor]

Arbitrary keyword arguments of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 {}

Returns:

Name Type Description
AsyncTask AsyncTask

An asynchronous task object representing the neural network execution. Call its wait method to block until the execution finishes and obtain the outputs.

async_run(*input_args, **input_kwargs) async

Run the compiled neural network asynchronously and returns the result awaitably.

This method is a coroutine that can be used with the await keyword to asynchronously run the neural network and retrieve the result.

Parameters:

Name Type Description Default
*input_args Union[ndarray, Tensor]

Variable length argument list of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 ()
**input_kwargs Union[ndarray, Tensor]

Arbitrary keyword arguments of input tensors. Each argument should be either a numpy.ndarray or a torch.Tensor.

 {}

Returns:

Type Description
Union[ndarray, Tensor, List[Union[ndarray, Tensor]]]

The output tensor(s) of the neural network. The return type depends on the network's architecture and can be either a single tensor or a list of tensors. The tensor type (numpy.ndarray or torch.Tensor) is determined by the tensor_type provided during the AsyncRuntime object's initialization.

AsyncTask

A handle for one asynchronous execution, returned by AsyncRuntime.run.

wait(timeout=None)

Wait for the asynchronous task to complete and returns the result.

This method blocks the calling thread until the task is completed or the specified timeout is reached.

Parameters:

Name Type Description Default
timeout float

The maximum amount of time (in seconds) to wait for the task to complete. If None, the method will wait indefinitely until the task is completed. Defaults to None.

 None

Returns:

Type Description
Union[ndarray, Tensor, List[Union[ndarray, Tensor]]]

The output tensor(s) of the neural network. The return depends on the network's architecture and can be either a single tensor or a list of tensors. The tensor type (numpy.ndarray or torch.Tensor) is determined by the tensor_type provided during the AsyncRuntime object's initialization.

Raises:

Type Description
RuntimeError

If the task has already returned its result.

utility API

DeviceProperties

Static properties of an NPU, as returned by get_device_properties.

Attributes:

Name Type Description
name str

Name of the NPU, e.g. "RBLN-CA25".
total_memory int

Total device memory in bytes (memory_per_chiplet * num_chiplet).
memory_per_chiplet int

Memory of one chiplet in bytes.
num_chiplet int

Number of chiplets in the NPU.

ChipletMemoryInfo

User DRAM pool of one chiplet, in bytes.

Each chiplet allocates from its own pool, so a physically contiguous buffer is bounded by one chiplet's largest_free, not by the device-wide free total.

Attributes:

Name Type Description
total int

Size of the pool.
used int

Bytes in use.
free int

Bytes free.
largest_free int

Largest free contiguous region.
largest_free_huge int

Largest free contiguous region at the huge granule, or 0 where the part has none.

DeviceMemoryInfo

Device-wide user DRAM usage, in bytes, as returned by get_device_memory_info.

Attributes:

Name Type Description
total int

Total allocatable memory, summed over chiplets.
used int

Bytes in use, summed over chiplets.
free int

Bytes free, summed over chiplets.
granularity int

Allocation granule in bytes.
huge_granularity int

Huge allocation granule in bytes.
chiplets List[ChipletMemoryInfo]

Per-chiplet usage.

npu_is_available(device_id=0)

Return a bool indicating whether the RBLN NPU is currently available.

Parameters:

Name Type Description Default
device_id int

Index of the NPU. Defaults to 0.

 0

Returns:

Name Type Description
bool bool

Whether the RBLN NPU is currently available.

get_npu_name(device_id=0)

Return the name of the RBLN NPU.

Parameters:

Name Type Description Default
device_id int

Index of the NPU. Defaults to 0.

 0

Returns:

Name Type Description
str str

The name of the NPU, e.g. "RBLN-CA12".

get_num_threads()

Return the number of host threads the runtime uses.

Returns:

Name Type Description
int int

The current thread count.

set_num_threads(num_threads)

Set the number of host threads the runtime uses.

Parameters:

Name Type Description Default
num_threads int

The thread count to use.

 required

search_num_threads(runtime, max_iterations=200)

Benchmark different thread counts and prints the average execution time for each configuration.

This function tests various thread counts by adjusting the num_threads property of the given runtime instance. For each configuration, it measures the average host execution time over a specified number of iterations and prints the results.

Parameters:

Name Type Description Default
runtime Runtime

The runtime instance whose thread count will be tested.

 required
max_iterations int

The number of iterations to run for each thread count. A higher value provides more accurate results. Defaults to 200.

 200
Example 
1
2
3
4
import rebel
from rebel.core.tools import search_num_threads
runtime = rebel.Runtime("model.rbln")
search_num_threads(runtime)
Output 
1
2
3
4
5
INFO [rebel-compiler] Testing with 1 thread.
INFO [rebel-compiler] 200 runs: Average execution time = 143.10 µs
INFO [rebel-compiler] Testing with 2 threads.
INFO [rebel-compiler] 200 runs: Average execution time = 80.15 µs
...

device_count()

Return the number of NPUs available.

Returns:

Name Type Description
int int

The number of NPUs available.

get_device_properties(device_id=0)

Return the static properties the NPU reports about itself.

Parameters:

Name Type Description Default
device_id int

Index of the NPU. Defaults to 0.

 0

Returns:

Type Description
Optional[DeviceProperties]

DeviceProperties or None: The properties of the NPU, or None when no NPU is found at that index.

get_device_memory_info(device_id=0)

Return the device-wide DRAM usage of the NPU, per chiplet and in total.

This needs no runtime and can be called before a model is loaded. The result is a reading, not a reservation: another process may allocate between this call and yours.

Parameters:

Name Type Description Default
device_id int

Index of the NPU. Defaults to 0.

 0

Returns:

Type Description
Optional[DeviceMemoryInfo]

DeviceMemoryInfo or None: The memory usage of the NPU, or None when the NPU is not available or the installed driver does not provide the query.

manual_seed(seed)

Set the manual seed of the RBLN Runtime.

Parameters:

Name Type Description Default
seed int

The seed to set.

 required