[2.0] [BACKPORT] of [1.x][FEATURE] CUDA graphs support (#19142)

[DickJC123]

This PR ports the CUDA Graphs support (by default off, enabled by MXNET_ENABLE_CUDA_GRAPHS=1) to 2.0. For gluon models with static_shape and static_alloc both true, enabling CUDA Graphs can significantly reduce CPU overheads, leading to end-to-end perf improvements. See PR #19142 for additional details.

[mxnet-bot]

Hey @DickJC123 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [windows-gpu, edge, centos-gpu, sanity, windows-cpu, unix-gpu, website, unix-cpu, centos-cpu, miscellaneous, clang]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[chinakook]

@DickJC123 Hi, guru. I've tried to test your commit on my faster rcnn, and it's great. It really get 17% performance increase on inferencing, but It can't be work with multi-thread such as one model shared with multiple threads. It will eat all the GPU memory gradually.

[leezu]

For 2.0, should cuda graphs be enabled by default?

Is there any update on "I'm still exploring the ways of automatically testing newly added operators in order for the feature to be able to be on by default, but I do not consider this the scope of this PR, as v1.x branch is not really supposed to get many more operators (I will do that in the PR to master). Generally this would involve testing operators with hybridize(static_alloc=True, static_shape=True) (which generally should be tested much more as right now testing of this functionality is really limited, even though it is widely used)." @ptrendx #19142 (comment)?

[chinakook]

@DickJC123 Hi, guru. I've tried to test your commit on my faster rcnn, and it's great. It really get 17% performance increase on inferencing, but It can't be work with multi-thread such as one model shared with multiple threads. It will eat all the GPU memory gradually.

I've tried compile mxnet with cuda 11.6 and the latest cudnn version. There are no bugs with cuda graphs.

[ptrendx]

LGTM, left a few questions.

[DickJC123]

Merging now based on the approval from @ptrendx, our successful experience with CUDA Graphs on the 1.x branch, and the notion that the CUDA Graphs is off by default (enabled via MXNET_ENABLE_CUDA_GRAPHS=1).