콘텐츠로 이동

에러 코드

컴파일 에러 코드

이 섹션에서는 RBLN 컴파일러 및 optimum-rbln를 포함한 RBLN SDK 사용 시 발생할 수 있는 컴파일 에러 코드를 설명합니다. 이러한 에러를 이해하면 컴파일 과정 중 문제를 해결하는 데 도움이 됩니다. RBLN 컴파일러와 RSD 기술보고서를 참고하면 이 에러 코드 페이지를 더 잘 이해하는 데 도움이 됩니다.

모델 컴파일은 두 단계로 나뉘며, 각 단계는 특정 하위 단계로 구성됩니다.

  • Computation Graph Generation: 모델을 중간 표현(intermediate representation, IR)으로 변환
    • Device Graph Conversion
    • Operation (Op) Fusion
  • Computation Graph Optimization: IR을 리벨리온 NPU 실행에 최적화
    • Multi-Device Splitting
    • Op Splitting and Grouping
    • Op Tiling
    • Op Scheduling
    • Bufferization
    • Memory Allocation
    • Dependency Analysis
    • Command Scheduling
    • Code Generation

아래는 각 에러 코드와 관련 컴파일 단계, 그리고 간결한 설명을 보여주는 테이블입니다. 명확한 이해를 위해 영문으로 작성되어 있습니다:

Error Code Related Compile Step Description

추론 에러 코드

이 섹션은 추론 과정에서 발생할 수 있는 에러 코드, 즉 컴파일된 모델(.rbln 파일)을 로딩하거나 추론을 수행하면서 발생할 수 있는 에러들을 설명합니다. 각 에러 코드의 접두사는 에러가 발생한 단계를 나타냅니다.

  • LOADING_: .rbln 파일을 읽어오는 도중 오류가 발생한 경우. 파일을 읽어오는 과정에서 문제가 발생했음을 나타냅니다. rebel-compiler를 최신 버전으로 업그레이드하고 모델을 다시 컴파일하는 것을 권장합니다.
  • INIT_: 모델을 초기화하는 과정에서 오류가 발생한 경우.
  • RUN_: 추론 실행 도중 오류가 발생한 경우.
  • SYS_: NPU 장치 또는 드라이버가 시스템 레벨 오류를 보고한 경우. 메시지 끝에 raw system code가 (code: N) 형태로 붙습니다.
  • RCCL_: multi-device 추론을 위한 RBLN-CCL 통신 초기화 도중 오류가 발생한 경우.

아래는 추론 과정에서 발생할 수 있는 에러 코드를 정리한 테이블입니다. 명확한 이해를 위해 영문으로 작성되어 있습니다:

Error Code Description Possible Solutions
INVALID_ARGUMENT An invalid argument was provided.
LOADING_FAILED Unexpected error occurred while loading the compiled model (.rbln file). Try re-compiling the model with the latest version of the compiler.
LOADING_FILE_NOT_FOUND The file does not exist at the specified path. Make sure that you gave the correct path to the .rbln file.
LOADING_INVALID_FILE
  • 103: The file is not a valid .rbln file.
  • 106: The contents of the model seem inconsistent or invalid.
  • 103: Make sure that you gave the correct path to the .rbln file or try re-compiling the model with the latest version of the compiler.
  • 106: Try re-compiling the model with the latest version of the compiler.
LOADING_UNEXPECTED_EOF The file is not a valid .rbln file. The .rbln file may be truncated. Make sure that you gave the correct path to the .rbln file or try re-compiling the model with the latest version of the compiler.
LOADING_INVALID_VERSION The compiled model is compiled with an incompatible compiler. Try re-compiling the model with the latest version of the compiler or update your runtime to the latest version.
LOADING_INVALID_CHECKSUM Failed to verify the checksum of the compiled model. The compiled model may be corrupted. Try re-compiling the model with the latest version of the compiler.
LOADING_INVALID_SHARED_OBJ The file may be corrupted or the target architecture of the compiled model does not match to your current environment. Try re-compiling the model with the latest version of the compiler for the target architecture.
INIT_INTERNAL Unexpected error occurred while initializing the device for the model.
INIT_INVALID_ARGUMENT An invalid argument was provided during initialization.
INIT_ALREADY_CREATED A runtime has already been created for the compiled model. Try creating a runtime on a different NPU or reusing the existing one.
INIT_FAILED_LOADING_SHARED_OBJ Failed to load the compiled host operator modules.
INIT_SHARED_OBJ_NO_SYMBOL Failed to initialize the host operator due to a missing symbol.
INIT_MEM_ALLOC_FAILED Failed to allocate device memory.
RUN_INTERNAL Unexpected error occurred while running the inference.
RUN_ERROR_FROM_OTHER_THREAD This error may occur when using async runtime. An error occurred from another thread and thus the run was aborted.
RUN_INVALID_NVME_CHUNK The NVMe chunk handle is invalid (already freed or never allocated).
SYS_ERROR An unexpected system-level error has occurred.
SYS_BUSY The operation is still in progress, but a timeout occurred due to the user-specified timeout. Try again with a longer timeout. Please refer to the timeout parameter of Runtime class.
SYS_KERNEL_TIMEOUT The operation on the device exceeded the kernel-specified timeout (default is 6 seconds).
SYS_TASK_ABORTED An error occurred during the execution of a task on the device.
SYS_UNKNOWN The job failed with an error the driver could not classify. Typically the owning process is exiting, or the device failed to recover from a reset. Check the device status with rbln-smi.
SYS_ENOENT No enabled system device.
SYS_ESRCH No such process.
SYS_EINTR Interrupted system call.
SYS_EIO I/O error.
SYS_ENXIO No such device or address.
SYS_ENOMEM Out of memory.
SYS_EBUSY Lack of device memory.
SYS_ENODEV No such device.
SYS_ENOSPC No space left on device.
SYS_EPIPE Broken pipe.
SYS_ECANCELLED Operation canceled.
SYS_ETIMEDOUT The operation timed out. The device may be unresponsive or under reset, or an RBLN-CCL collective may be waiting on a peer rank that stopped responding. For multi-device inference, make sure that all ranks are still running. Check the device status with rbln-smi.
SYS_ECONNREFUSED The device was torn down or failed to recover from a reset. In-flight work on this context was terminated and will not be resubmitted. Check the device status with rbln-smi. Re-create the runtime after the device recovers.
SYS_ENOTSUP The operation is not supported by this device, driver, or collective configuration. Make sure that the RBLN driver version is compatible with your RBLN SDK version.
SYS_EPERM The operation was rejected. The context may already be destroyed, the device may be in reset, or an RBLN-CCL collective may require a device capability this device does not have.
SYS_ESHUTDOWN The context or device is shutting down and its resources are being reclaimed.
SYS_EAGAIN The device is busy or recovering. Retry the operation.
RCCL_GET_UNIQUE_ID_FAILED RBLN-CCL unique ID generation failed. Set RBLN_ROOT_IP and RBLN_LOCAL_IP before launching (single node: both 127.0.0.1; multi node: the root node IP and the local node IP).
RCCL_COMM_INIT_RANK_FAILED RBLN-CCL communication initialization failed.
RCCL_ADDRESS_IN_USE RBLN-CCL address in use. Specify a different port with the RCCL_INIT_BASE_PORT and/or RCCL_BASE_RDMA_PORT environment variables, or enable automatic port selection.
RCCL_COMM_SPLIT_FAILED RBLN-CCL communication subgroup split failed.