Skip to content

Error Codes

Compile-time error codes

This section details compile-time error codes you may encounter when using the RBLN SDK, specifically with the rebel-compiler and optimum-rbln tools. These errors occur during model compilation for the RBLN NPU, and understanding them aids in diagnosing and fixing issues. For deeper insight into the compilation process, refer to the RBLN Compiler and RSD white papers.

The model compilation workflow consists of two primary stages, each encompassing specific steps:

  • Computation Graph Generation: Converts the model into an intermediate representation (IR). Steps include:
    • Device Graph Conversion
    • Operation (Op) Fusion
  • Computation Graph Optimization: Optimizes the IR for RBLN NPU execution. Steps include:
    • Multi-Device Splitting
    • Op Splitting and Grouping
    • Op Tiling
    • Op Scheduling
    • Bufferization
    • Memory Allocation
    • Dependency Analysis
    • Command Scheduling
    • Code Generation

The table below lists error codes, their associated compile steps, and descriptions:

Error Code Related Compile Step Description

Inference-time error codes

This section lists the inference-time error codes that you might encounter when loading a compiled model (.rbln file) or running inference. The prefix of each error code indicates the stage at which the error occurred.

  • LOADING_: The error occurred while loading the .rbln file. This may indicate an issue while reading the file. Consider upgrading your rebel-compiler to the latest version and recompiling your model.
  • INIT_: The error occurred while initializing the model.
  • RUN_: The error occurred while running inference.
  • SYS_: A system-level error was reported by the NPU device or its driver. The raw system code is appended to the message as (code: N).
  • RCCL_: The error occurred while initializing RBLN-CCL communication for multi-device inference.
Error Code Description Possible Solutions
INVALID_ARGUMENT An invalid argument was provided.
LOADING_FAILED Unexpected error occurred while loading the compiled model (.rbln file). Try re-compiling the model with the latest version of the compiler.
LOADING_FILE_NOT_FOUND The file does not exist at the specified path. Make sure that you gave the correct path to the .rbln file.
LOADING_INVALID_FILE
  • 103: The file is not a valid .rbln file.
  • 106: The contents of the model seem inconsistent or invalid.
  • 103: Make sure that you gave the correct path to the .rbln file or try re-compiling the model with the latest version of the compiler.
  • 106: Try re-compiling the model with the latest version of the compiler.
LOADING_UNEXPECTED_EOF The file is not a valid .rbln file. The .rbln file may be truncated. Make sure that you gave the correct path to the .rbln file or try re-compiling the model with the latest version of the compiler.
LOADING_INVALID_VERSION The compiled model is compiled with an incompatible compiler. Try re-compiling the model with the latest version of the compiler or update your runtime to the latest version.
LOADING_INVALID_CHECKSUM Failed to verify the checksum of the compiled model. The compiled model may be corrupted. Try re-compiling the model with the latest version of the compiler.
LOADING_INVALID_SHARED_OBJ The file may be corrupted or the target architecture of the compiled model does not match to your current environment. Try re-compiling the model with the latest version of the compiler for the target architecture.
INIT_INTERNAL Unexpected error occurred while initializing the device for the model.
INIT_INVALID_ARGUMENT An invalid argument was provided during initialization.
INIT_ALREADY_CREATED A runtime has already been created for the compiled model. Try creating a runtime on a different NPU or reusing the existing one.
INIT_FAILED_LOADING_SHARED_OBJ Failed to load the compiled host operator modules.
INIT_SHARED_OBJ_NO_SYMBOL Failed to initialize the host operator due to a missing symbol.
INIT_MEM_ALLOC_FAILED Failed to allocate device memory.
RUN_INTERNAL Unexpected error occurred while running the inference.
RUN_ERROR_FROM_OTHER_THREAD This error may occur when using async runtime. An error occurred from another thread and the run was thus aborted.
RUN_INVALID_NVME_CHUNK The NVMe chunk handle is invalid (already freed or never allocated).
SYS_ERROR An unexpected system-level error has occurred.
SYS_BUSY The operation is still in progress, but a timeout occurred due to the user-specified timeout. Try again with a longer timeout. Refer to the timeout parameter of Runtime class.
SYS_KERNEL_TIMEOUT The operation on the device exceeded the kernel-specified timeout. (Default is 6 seconds.)
SYS_TASK_ABORTED An error occurred during the execution of a task on the device.
SYS_UNKNOWN The job failed with an error the driver could not classify. Typically the owning process is exiting, or the device failed to recover from a reset. Check the device status with rbln-smi.
SYS_ENOENT No enabled system device.
SYS_ESRCH No such process.
SYS_EINTR Interrupted system call.
SYS_EIO I/O error.
SYS_ENXIO No such device or address.
SYS_ENOMEM Out of memory.
SYS_EBUSY Lack of device memory.
SYS_ENODEV No such device.
SYS_ENOSPC No space left on device.
SYS_EPIPE Broken pipe.
SYS_ECANCELLED Operation cancelled.
SYS_ETIMEDOUT The operation timed out. The device may be unresponsive or under reset, or an RBLN-CCL collective may be waiting on a peer rank that stopped responding. For multi-device inference, make sure that all ranks are still running. Check the device status with rbln-smi.
SYS_ECONNREFUSED The device was torn down or failed to recover from a reset. In-flight work on this context was terminated and will not be resubmitted. Check the device status with rbln-smi. Re-create the runtime after the device recovers.
SYS_ENOTSUP The operation is not supported by this device, driver, or collective configuration. Make sure that the RBLN driver version is compatible with your RBLN SDK version.
SYS_EPERM The operation was rejected. The context may already be destroyed, the device may be in reset, or an RBLN-CCL collective may require a device capability this device does not have.
SYS_ESHUTDOWN The context or device is shutting down and its resources are being reclaimed.
SYS_EAGAIN The device is busy or recovering. Retry the operation.
RCCL_GET_UNIQUE_ID_FAILED RBLN-CCL unique ID generation failed. Set RBLN_ROOT_IP and RBLN_LOCAL_IP before launching (single node: both 127.0.0.1; multi node: the root node IP and the local node IP).
RCCL_COMM_INIT_RANK_FAILED RBLN-CCL communication initialization failed.
RCCL_ADDRESS_IN_USE RBLN-CCL address in use. Specify a different port with the RCCL_INIT_BASE_PORT and/or RCCL_BASE_RDMA_PORT environment variables, or enable automatic port selection.
RCCL_COMM_SPLIT_FAILED RBLN-CCL communication subgroup split failed.