Always getting "Failed to create CUDAExecutionProvider" #11092

fijipants · 2022-04-02T03:52:54Z

Describe the bug

When I try to create InferenceSession in Python with providers=['CUDAExecutionProvider'], I get the warning:

2022-04-01 22:45:36.716353289 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:535 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.

And my CPU usage shoots up, while my GPU usage stays at 0.

Urgency

Not urgent.

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
ONNX Runtime installed from (source or binary): binary
ONNX Runtime version: 1.10.0
Python version: 3.7.13
Visual Studio version (if applicable):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: CUDA 11.4 / cuDNN 8.2.4
GPU model and memory: RTX 3090 24GB

To Reproduce

Download and run any model, e.g. here's one from PyTorch's ONNX example:

super_resolution.zip

pip install onnxruntime-gpu==1.10.0

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession('../../Downloads/super_resolution.onnx', providers=['CUDAExecutionProvider'])
sess.run(None, {
    sess.get_inputs()[0].name: np.random.rand(1, 1, 224, 224).astype(np.float32)
})

Expected behavior

It should work, or at least print out a more informative warning.

Screenshots

Additional context

The text was updated successfully, but these errors were encountered:

ytaous · 2022-04-02T07:32:35Z

Hi, your code works for me on local dev box.
Can you please check nvidia-smi and see if gpus are visible?

fijipants · 2022-04-02T15:21:14Z

@ytaous Yes,

fijipants@FPSERVER:~$ nvidia-smi
Sat Apr  2 11:20:28 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   28C    P8    18W / 370W |     10MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 30%   25C    P8     9W / 350W |     10MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1347      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A      2133      G   /usr/lib/xorg/Xorg                  4MiB |
|    1   N/A  N/A      1347      G   /usr/lib/xorg/Xorg                  4MiB |
|    1   N/A  N/A      2133      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

fijipants · 2022-04-02T16:47:31Z

I was able to get it working inside of a Docker container nvidia/cuda:11.4.3-cudnn8-devel-ubuntu20.04.

Then I tried it again locally, and it worked! The only thing I did locally was apt install nvidia-container-toolkit after adding the sources.

Should that be added as a requirement?

skottmckay · 2022-04-05T07:49:41Z

Not sure nvidia-container-toolkit should be added as a requirement given we don't need to install that in our CIs and it's focused on building in a docker container (i.e. specific to that setup and not building with CUDA in general).

Often the CUDA EP not loading is an issue with cudnn not being installed correctly, leading to missing required libraries. These instructions need to be followed exactly to get everything in the right places: https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html

yaliqin · 2022-08-20T02:24:52Z

@fijipants can you share the final workable Dockerfile? I am struggling on this problem for 2 days.

serdarildercaglar · 2022-11-22T10:05:34Z

I have been facing the same problem for 3 days. And I couldn't find any solution. People have asked many questions and opened threads about similar issues. unfortunately no answer from Microsoft Team. Am I wrong?

oguz-hanoglu · 2023-01-18T10:39:32Z

https://stackoverflow.com/questions/75267445/why-does-onnxruntime-fail-to-create-cudaexecutionprovider-in-linuxubuntu-20/75267493#75267493
solved my problem.

BotScutters · 2023-01-24T23:32:35Z

I hate that this fixed it, but yes, importing torch before onnxruntime fixed it for me as well.

ssube · 2023-03-01T15:00:22Z

I ran into this as well and had to look through a few different files before I found the imports at fault. Is the import order documented anywhere?

I'm now importing ORT through a module that also imports torch:

# this file exists to make sure torch is always imported before onnxruntime
# to work around /~https://github.com/microsoft/onnxruntime/issues/11092

import torch  # NOQA
from onnxruntime import *  # NOQA

LeiWang1999 · 2023-03-22T12:02:11Z

Got same issue under GTX A6000 and my onnx runtime version is 1.14.1 and cuda version is 12.0, any updates?

w2ex · 2023-03-22T16:28:22Z

Having the same issue also, GPU is A100, onnxruntime-gpu 1.14.1, CUDA 12.0. Importing torch before onnxruntime does not solve the issue. I don't know how to solve this and it is quite annoying. If someone has an idea...

LeiWang1999 · 2023-03-26T09:14:14Z

Please CC @microsoft/onnxruntime @microsoft/onnxruntime-admin

skottmckay · 2023-03-27T08:53:12Z

Please make sure both CUDA and CUDNN are fully installed.

The reason it works if the GPU torch package is loaded first is because torch includes the CUDA binaries in the python package. There is no actual dependency between ORT and torch, so this workaround is covering for an invalid system wide install of CUDA + CUDNN.

I started with a clean Ubuntu 22.04 WSL install. I saw the error from the attempted usage of the CUDA EP until CUDA and CUDNN were fully installed. I installed CUDA and CUDNN using the NVIDIA packages.

I used this script to see which libraries were loaded (adjust the model path to something valid):

import ctypes
import onnxruntime as ort

def get_loaded_libraries():
    from ctypes.util import find_library

    class _dl_phdr_info(ctypes.Structure):
        _fields_ = [
            ("dlpi_addr", ctypes.c_uint64),
            ("dlpi_name", ctypes.c_char_p),
            ("dlpi_phdr", ctypes.c_void_p),
            ("dlpi_phnum", ctypes.c_uint32),
        ]

    def match_library_callback(info, size, data):
        # Get the path of the current library
        filepath = info.contents.dlpi_name
        if filepath:
            filepath = filepath.decode("utf-8")
            print(filepath)

        return 0

    c_func_signature = ctypes.CFUNCTYPE(ctypes.c_int,  # Return type
                                        ctypes.POINTER(_dl_phdr_info),
                                        ctypes.c_size_t,
                                        ctypes.c_char_p,)

    c_match_library_callback = c_func_signature(match_library_callback)
    data = ctypes.c_char_p(b"")

    dl_iterate_phdr = ctypes.CDLL('libc.so.6').dl_iterate_phdr
    dl_iterate_phdr(c_match_library_callback, data)


get_loaded_libraries()
ort_session = ort.InferenceSession("mnist.onnx", providers=["CUDAExecutionProvider"])

Once CUDA and CUDNN were installed fully I see the following in the output from the script. List trimmed to relevant libraries.

/lib/x86_64-linux-gnu/libcudnn.so.8
/usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.11
/usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.so
/usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.11
/usr/local/cuda/targets/x86_64-linux/lib/libcurand.so.10
/usr/local/cuda/targets/x86_64-linux/lib/libcufft.so.10
/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.0
/lib/x86_64-linux-gnu/libz.so.1

The loaded libraries match the CUDA dependencies the ORT library has here, along with the CUDNN requirement for libz to have been installed.

If it is using CUDA binaries from the torch install you'll see the python site-packages directory in the path.

e.g. from a user local install of torch

/home/me/.local/lib/python3.10/site-packages/torch/lib/../../nvidia/curand/lib/libcurand.so.10
/home/me/.local/lib/python3.10/site-packages/torch/lib/../../nvidia/cufft/lib/libcufft.so.10
/home/me/.local/lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublas.so.11
/home/me/.local/lib/python3.10/site-packages/torch/lib/../../nvidia/cublas/lib/libcublasLt.so.11
/home/me/.local/lib/python3.10/site-packages/torch/lib/../../nvidia/cuda_runtime/lib/libcudart.so.11.0
/home/me/.local/lib/python3.10/site-packages/torch/lib/../../nvidia/cudnn/lib/libcudnn.so.8

LeiWang1999 · 2023-03-27T12:39:22Z

I worked with nvidia ngc docker container nvcr.io/nvidia/pytorch:23.01-py3 , which have all the envrioment installed initially, but still failed.

skottmckay · 2023-03-27T22:23:17Z

I worked with nvidia ngc docker container nvcr.io/nvidia/pytorch:23.01-py3 , which have all the envrioment installed initially, but still failed.

I'd suggest running the script to see which libraries are not being loaded as expected. The location of those libraries should be listed in the output from cat /etc/ld.so.conf.d/* (at least on Ubuntu - not sure if other linux distributions differ).

There's also a known issue listed at the bottom of this page that mentions having to manually set LD_LIBRARY_PATH to resolve a problem with cudnn. Not sure if that is potentially a factor.

ORT doesn't officially support CUDA 12 yet either, so it may be better to try the 22.12 version of the container.

LeiWang1999 · 2023-03-28T09:28:46Z

Thanks but the output is, now I fall back into 22.06 and it works for me now ... maybe ort doesn't officially support CUDA 12 even while they announced at the latest release note 😆 :

linux-vdso.so.1
/lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libpthread.so.0
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libutil.so.1
/lib/x86_64-linux-gnu/libm.so.6
/lib/x86_64-linux-gnu/libexpat.so.1
/lib/x86_64-linux-gnu/libz.so.1
/lib64/ld-linux-x86-64.so.2
/usr/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so
/lib/x86_64-linux-gnu/libffi.so.7
/usr/local/cuda-11/targets/x86_64-linux/lib/libcublasLt.so.11
/lib/x86_64-linux-gnu/librt.so.1
/lib/x86_64-linux-gnu/libgcc_s.so.1
/usr/local/cuda/compat/lib/libcuda.so.1
/usr/local/cuda-11/targets/x86_64-linux/lib/libnvrtc.11
/usr/local/cuda-11/targets/x86_64-linux/lib/libcublas.so.11
/lib/x86_64-linux-gnu/libcudnn.so.8
/lib/x86_64-linux-gnu/libstdc++.so.6
/usr/local/cuda/targets/x86_64-linux/lib/libcurand.so.10
/usr/local/cuda-11/targets/x86_64-linux/lib/libcudart.so.11.0
/lib/x86_64-linux-gnu/libnvinfer.so.8
/lib/x86_64-linux-gnu/libnvinfer_plugin.so.8
/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/onnxruntime_pybind11_state.cpython-38-x86_64-linux-gnu.so
/usr/lib/python3.8/lib-dynload/_json.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_umath.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/core/../../numpy.libs/libopenblas64_p-r0-15028c96.3.21.so
/usr/local/lib/python3.8/dist-packages/numpy/core/../../numpy.libs/libgfortran-040039e1.so.5.0.0
/usr/local/lib/python3.8/dist-packages/numpy/core/../../numpy.libs/libquadmath-96973f99.so.0.0.0
/usr/lib/python3.8/lib-dynload/_contextvars.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/core/_multiarray_tests.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/linalg/_umath_linalg.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/fft/_pocketfft_internal.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/random/mtrand.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/random/bit_generator.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/random/_common.cpython-38-x86_64-linux-gnu.so
/usr/lib/python3.8/lib-dynload/_hashlib.cpython-38-x86_64-linux-gnu.so
/lib/x86_64-linux-gnu/libcrypto.so.1.1
/usr/local/lib/python3.8/dist-packages/numpy/random/_bounded_integers.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/random/_mt19937.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/random/_philox.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/random/_pcg64.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/random/_sfc64.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/numpy/random/_generator.cpython-38-x86_64-linux-gnu.so
/usr/lib/python3.8/lib-dynload/_opcode.cpython-38-x86_64-linux-gnu.so
/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/libonnxruntime_providers_shared.so
/usr/lib/python3.8/lib-dynload/_bz2.cpython-38-x86_64-linux-gnu.so
/lib/x86_64-linux-gnu/libbz2.so.1.0
/usr/lib/python3.8/lib-dynload/_lzma.cpython-38-x86_64-linux-gnu.so
/lib/x86_64-linux-gnu/liblzma.so.5
2023-03-28 00:26:00.089302146 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:541 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.

pranavsharma · 2023-03-28T16:56:59Z

CUDA 12 is not fully supported. We noted build compatibility in the 1.14 release notes, however it was partial (only Linux) as we found issues with Windows builds later.

skottmckay · 2023-03-30T06:36:42Z

FWIW printing out the loaded libraries can be done with this function on multiple platforms if you install psutil, which is much simpler and more flexible than the above version that used dl_iterate_phdr.

Note that on Windows the CUDA libraries do not get loaded on import onnxruntime unlike on Linux.

def get_loaded_libraries():
    import os
    import psutil
    p = psutil.Process(os.getpid())
    for lib in p.memory_maps():
      print(lib.path)

dcdethan · 2023-05-09T02:44:11Z

First ,you should check your onnxruntime-gpu version ,cuda version and cudnn version in this link 'https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html'
Second,use 'nvcc -V' and 'cat /usr/local/cuda/include/cudnn_version.h' make sure your env is usefull.
Last, import torch is necessary,the code as is follows:
import torch
import onnxruntime as rt
providers = ['CUDAExecutionProvider', 'CPUExecutionProvider']
session = rt.InferenceSession("yolo.onnx", providers=providers)

ajinkya-algo · 2023-05-09T12:30:24Z

Having the same issue also, GPU is A100, onnxruntime-gpu 1.14.1, CUDA 12.0. Importing torch before onnxruntime does not solve the issue. I don't know how to solve this and it is quite annoying. If someone has an idea...

Have you resolved your issue?
Because i'm facing the same issue

tianleiwu · 2023-05-09T16:48:57Z

GPU is A100, onnxruntime-gpu 1.14.1, CUDA 12.0

@ajinkya-algo, Need install CUDA 11.* if you want to use onnxruntime-gpu 1.14.1, another solution is to build onnxruntime-gpu from source for CUDA 12.0.

kopyl · 2023-06-23T08:55:38Z

@BotScutters thank you very much. Importing torch before onnxruntime helped.
Do you know why by the way? And how to avoid it? Torch is really huge in size, don't really want to bring it into my Docker image...

tianleiwu · 2023-06-23T17:30:39Z

@kopyl, onnxruntime does not depend on torch. You can just install onnxruntime-gpu with compatible version of CUDA and cuDNN in docker image.

For Ubuntu, it could be like the following (cuda 11.6 to 11.8, cuDNN 8.5 to 8.7 shall be fine):

pip3 install onnxruntime-gpu==1.15.1 cuda-python==11.8.0 nvidia-cudnn-cu11==8.5.0.96

kopyl · 2023-06-23T21:16:27Z

@tianleiwu thank you. Can I run it on Cuda 12?

tianleiwu · 2023-06-23T23:11:04Z

@kopyl, no, the official package does not support cuda 12. However, you can build from source to run in cuda 12.

fireinice · 2023-07-14T22:14:40Z

For anyone who may trap in the same problem, I record all I done for a week to bring onnxruntime-gpu up in a container on debian system as below:

install nvidia drive on debian from nvidia, follow instructions https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#network-repo-installation-for-debian
for the system, the cuda-tookit or nvidia-cuda is not necessary. Only add nvidia apt source and install nvidia-driver

apt update && apt install nvidia-driver

check driver status with nvidia-smi, It should look like:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+

the driver veresion and cuda version should both not empty.
(I tried nvidia-tesla-470-driver from debian apt source firstly, but it seems not working. the cuda version showed empty and ort throw exceptions complained the driver version too low

install nvidia-container-tookit follow instructions from https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
and then I could bring up python3.9 docker image with docker-compose config

version: "2.2"
services:
  modelsrv:
    image: python:3.9.17-slim-bullseye
    container_name: modelsrv
    command: sleep 100000d
    restart: always
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

the configuration is from https://docs.docker.com/compose/gpu-support/.

and in the container, the command nvidia-smi should output the same info as from the base system.

in the container install pip packages as below:

pip install nvidia-pyindex
pip install onnxruntime-gpu==1.15.1 nvidia-cublas==11.5.1.101 nvidia-cublas-cu117==11.10.1.25 nvidia-cuda-runtime-cu114==11.4.148 nvidia-cudnn==8.2.0.51 nvidia-cudnn-cu11==8.5.0.96 nvidia-cufft==10.4.2.58 nvidia-curand==10.2.4.58

and then export LD_LIBRARY_PATH as

export LD_LIBRARY_PATH=/usr/local/lib/python3.9/site-packages/nvidia/cublas/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cuda_runtime/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cudnn/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cufft/lib/:/usr/local/lib/python3.9/site-packages/nvidia/curand/lib/

And it works fine by now.

for tensorrt just install tensort and add tensorrt_libs to LD_LIBRARY_PATH:

pip install tensorrt

export LD_LIBRARY_PATH=/usr/local/lib/python3.9/site-packages/nvidia/cublas/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cuda_runtime/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cudnn/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cufft/lib/:/usr/local/lib/python3.9/site-packages/nvidia/curand/lib/:/usr/local/lib/python3.9/site-packages/tensorrt_libs/

Apisteftos · 2023-07-27T01:41:36Z

For anyone who may trap in the same problem, I record all I done for a week to bring onnxruntime-gpu up in a container on debian system as below:

install nvidia drive on debian from nvidia, follow instructions https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#network-repo-installation-for-debian for the system, the cuda-tookit or nvidia-cuda is not necessary. Only add nvidia apt source and install nvidia-driver
apt update && apt install nvidia-driver
check driver status with nvidia-smi, It should look like:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
the driver veresion and cuda version should both not empty. (I tried nvidia-tesla-470-driver from debian apt source firstly, but it seems not working. the cuda version showed empty and ort throw exceptions complained the driver version too low

install nvidia-container-tookit follow instructions from https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html and then I could bring up python3.9 docker image with docker-compose config
version: "2.2"
services:
  modelsrv:
    image: python:3.9.17-slim-bullseye
    container_name: modelsrv
    command: sleep 100000d
    restart: always
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
the configuration is from https://docs.docker.com/compose/gpu-support/.

and in the container, the command nvidia-smi should output the same info as from the base system.

in the container install pip packages as below:
pip install nvidia-pyindex
pip install onnxruntime-gpu==1.15.1 nvidia-cublas==11.5.1.101 nvidia-cublas-cu117==11.10.1.25 nvidia-cuda-runtime-cu114==11.4.148 nvidia-cudnn==8.2.0.51 nvidia-cudnn-cu11==8.5.0.96 nvidia-cufft==10.4.2.58 nvidia-curand==10.2.4.58
and then export LD_LIBRARY_PATH as
export LD_LIBRARY_PATH=/usr/local/lib/python3.9/site-packages/nvidia/cublas/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cuda_runtime/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cudnn/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cufft/lib/:/usr/local/lib/python3.9/site-packages/nvidia/curand/lib/
And it works fine by now.

for tensorrt just install tensort and add tensorrt_libs to LD_LIBRARY_PATH:
pip install tensorrt
export LD_LIBRARY_PATH=/usr/local/lib/python3.9/site-packages/nvidia/cublas/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cuda_runtime/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cudnn/lib/:/usr/local/lib/python3.9/site-packages/nvidia/cufft/lib/:/usr/local/lib/python3.9/site-packages/nvidia/curand/lib/:/usr/local/lib/python3.9/site-packages/tensorrt_libs/

Doesn't work

snnn · 2023-08-15T05:23:57Z

Once I remove patchelf from setup.py and resolve the issue #9754 , it will be much easier for you to know which shared library was missing.

avinash-218 · 2023-10-04T09:06:28Z

I hate that this fixed it, but yes, importing torch before onnxruntime fixed it for me as well.

yesss this worked for me as well. Any idea why this happens and how to solve the issue without installing torch in the environment?

skottmckay · 2023-10-05T22:29:43Z

Importing torch is NOT necessary UNLESS you have not installed the required CUDA libraries and setup the environment correctly.

The python torch GPU module directly includes all the CUDA libraries which are very large - hence the size of that package is over 2.5GB. When you import torch it updates the environment the script is running in to point to the CUDA libraries directly included in that package.

Downloading https://download.pytorch.org/whl/cu118/torch-2.1.0%2Bcu118-cp311-cp311-win_amd64.whl (2722.7 MB)

e.g. from windows showing some of the CUDA related libraries in the torch packages's lib directory

The ORT python package does not include the CUDA libraries, which is more efficient and flexible, but requires the user to install the required CUDA libraries and setup the environment correctly so that they can be found. Note that you need CUDA and cuDNN to be installed.

avinash-218 · 2023-10-19T14:29:19Z

I hate that this fixed it, but yes, importing torch before onnxruntime fixed it for me as well.

Can you please tell me whose cuda and cudnn is utilized by onnxruntime

if torch is imported before onnxruntime
if torch is imported after onnxruntime

tianleiwu · 2023-10-24T06:48:56Z

@avinash-218, you can try run a python script like the following in your machine:

import onnxruntime
import torch
import psutil
import os

session = onnxruntime.InferenceSession(
        "model.onnx", providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)

p = psutil.Process(os.getpid())
for lib in p.memory_maps():
   print(lib.path)

Then change the order of first two lines, and run again.

In Windows, I tested torch 2.0.1 and ORT 1.16.1, and it seems that import order does not matter. In both case, the cuda and cudnn from torch will be used. It is likely torch will load those DLLs during import, while ORT will delay loading cuda and cudnn until they are used.

In Linux, the result might be different so you will need to try it by yourself.

vadimkantorov · 2024-03-15T11:39:40Z

any update on official CUDA 12 builds support?

tianleiwu · 2024-03-15T17:18:45Z

@vadimkantorov, cuda 12 is supported. See https://onnxruntime.ai/docs/install/ for installation.

serdarildercaglar · 2024-03-28T09:16:32Z

This docker images works for me.

# Stage 1: Builder/Compiler
FROM nvidia/cuda:12.2.2-cudnn8-runtime-ubuntu22.04 AS builder
ARG DEBIAN_FRONTEND=noninteractive

# Install dependencies
# Update and install dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    python3 \
    python3-pip \
    python3-setuptools \
    python3-wheel \
    && rm -rf /var/lib/apt/lists/*

# Install Python virtual environment
RUN python3 -m pip install --upgrade pip setuptools




COPY requirements.txt /.
RUN pip3 install --no-cache-dir -r requirements.txt
RUN pip3 install onnxruntime-gpu==1.17.1 --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
COPY ./nltk_data /root/nltk_data

EXPOSE 8080

# CMD ["sh","-c", "jupyter lab --notebook-dir=/home/jovyan --ip=0.0.0.0 --no-browser --allow-root --port=8888 --NotebookApp.token='' --NotebookApp.password='' --NotebookApp.allow_origin='*' --NotebookApp.base_url=${NB_PREFIX}"]```

[requirements.txt](/~https://github.com/microsoft/onnxruntime/files/14786985/requirements.txt)

Arcitec · 2024-04-18T06:06:39Z

I hate that this fixed it, but yes, importing torch before onnxruntime fixed it for me as well.

Read this if you want to know why this issue happens:

The problem is that ONNX doesn't know how to search for CUDA. PyTorch knows how to search for it, and adds it to Python's internal path, so that ONNX can later find it.

The bug/issue is with ONNX library. I have coded a workaround here:

cubiq/ComfyUI_IPAdapter_plus#238

tcotte · 2025-02-12T13:05:41Z

@skottmckay provided the response to use onnx-runtime on CUDA.

I have written Python code to import CUDA binaries:
` def get_nvidia_binaries(nvidia_libs_path: str) -> None:

    import ctypes

    for i in os.listdir(nvidia_libs_path):
        if i.endswith('.dll'):
            try:
                ctypes.WinDLL(os.path.join(nvidia_libs_path, i))
                print(f'Imported: {i}')

            except:
                print(f'Not imported {i}')`

Use:
get_nvidia_binaries(nvidia_libs_path=os.path.join(os.environ['CUDA_PATH'], 'bin'))

weiji14 mentioned this issue Apr 15, 2022

Getting onnxruntime to work with CUDAExecutionProvider on gpu-pytorch container microsoft/planetary-computer-containers#33

Closed

yufenglee added the ep:CUDA issues related to the CUDA execution provider label Dec 6, 2022

tengerye mentioned this issue Mar 30, 2023

onnxruntime crashes on importing torch #15058

Closed

Apisteftos mentioned this issue Jun 11, 2023

Failed to create CUDAExecutionProvider #16128

Closed

tianleiwu mentioned this issue Jun 16, 2023

Segmentation fault when I import onnxruntime berofe torch #16388

Closed

This was referenced Jun 25, 2023

Only CPUExecutionProvider is loaded if not importing pytorch deepinsight/insightface#2344

Open

a hack to for onnxruntime to use cuda xaviviro/refacer#55

Merged

valentynbez mentioned this issue Aug 10, 2023

Remove torch dependency to run on GPU bioinf-mcb/Metagenomic-DeepFRI#51

Closed

amey-matroid mentioned this issue Sep 7, 2023

onnxruntime-gpu doesnt work with CUDA-12 and triton defaults to using CPUExecutionProvider triton-inference-server/server#6272

Closed

skottmckay mentioned this issue Mar 15, 2024

Microsoft.ML.OnnxRuntime.Gpu not working in MAUI project #14974

Open

hkhare42 mentioned this issue Jul 25, 2024

"Failed to create CUDAExecutionProvider" onnxruntime error on Google Colab roboflow/supervision#1401

Closed

1 task

Always getting "Failed to create CUDAExecutionProvider" #11092

Always getting "Failed to create CUDAExecutionProvider" #11092

Comments

fijipants commented Apr 2, 2022 • edited Loading

ytaous commented Apr 2, 2022

fijipants commented Apr 2, 2022 • edited Loading

fijipants commented Apr 2, 2022

skottmckay commented Apr 5, 2022

yaliqin commented Aug 20, 2022

serdarildercaglar commented Nov 22, 2022

oguz-hanoglu commented Jan 18, 2023 • edited Loading

BotScutters commented Jan 24, 2023

ssube commented Mar 1, 2023

LeiWang1999 commented Mar 22, 2023

w2ex commented Mar 22, 2023

LeiWang1999 commented Mar 26, 2023

skottmckay commented Mar 27, 2023

LeiWang1999 commented Mar 27, 2023

skottmckay commented Mar 27, 2023 • edited Loading

LeiWang1999 commented Mar 28, 2023

pranavsharma commented Mar 28, 2023

skottmckay commented Mar 30, 2023

dcdethan commented May 9, 2023

ajinkya-algo commented May 9, 2023 • edited Loading

tianleiwu commented May 9, 2023 • edited Loading

kopyl commented Jun 23, 2023

tianleiwu commented Jun 23, 2023 • edited Loading

kopyl commented Jun 23, 2023

tianleiwu commented Jun 23, 2023 • edited Loading

fireinice commented Jul 14, 2023 • edited Loading

Apisteftos commented Jul 27, 2023

snnn commented Aug 15, 2023

avinash-218 commented Oct 4, 2023

skottmckay commented Oct 5, 2023

avinash-218 commented Oct 19, 2023

tianleiwu commented Oct 24, 2023 • edited Loading

vadimkantorov commented Mar 15, 2024

tianleiwu commented Mar 15, 2024

serdarildercaglar commented Mar 28, 2024 • edited Loading

Arcitec commented Apr 18, 2024 • edited Loading

Read this if you want to know why this issue happens:

tcotte commented Feb 12, 2025 • edited Loading

fijipants commented Apr 2, 2022 •

edited

Loading

fijipants commented Apr 2, 2022 •

edited

Loading

oguz-hanoglu commented Jan 18, 2023 •

edited

Loading

skottmckay commented Mar 27, 2023 •

edited

Loading

ajinkya-algo commented May 9, 2023 •

edited

Loading

tianleiwu commented May 9, 2023 •

edited

Loading

tianleiwu commented Jun 23, 2023 •

edited

Loading

tianleiwu commented Jun 23, 2023 •

edited

Loading

fireinice commented Jul 14, 2023 •

edited

Loading

tianleiwu commented Oct 24, 2023 •

edited

Loading

serdarildercaglar commented Mar 28, 2024 •

edited

Loading

Arcitec commented Apr 18, 2024 •

edited

Loading

tcotte commented Feb 12, 2025 •

edited

Loading