Interleaved MHA for CPU path

[TaoLv]

Description

This PR fills the CPU counterpart of PR #16408.
Only FP32 is supported yet.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
• Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
• Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
• Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
• Code is well-documented:
• For user-facing API changes, API doc string has been updated.
• For new C++ functions in header files, their functionalities and arguments are documented.
• For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
• Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
• To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[TaoLv]

@eric-haibin-lin @pengzhao-intel @Caenorst Please help to review. I copied the UTs from test_operator_gpu.py to test_operator.py but didn't remove the original ones since it seems they have minimal cuda version requirement. Let me know if it's not appropriate. Thanks!

[eric-haibin-lin]

could you elaborate "it seems they have minimal cuda version requirement" and why we duplicate the test?

[TaoLv]

could you elaborate "it seems they have minimal cuda version requirement" and why we duplicate the test?

Because of this line: /~https://github.com/apache/incubator-mxnet/blob/master/tests/python/gpu/test_operator_gpu.py#L2863
But since the oldest cuda version in CI is 9.2 so I think it's safe to remove this check and copy tests to test_operator.py. I will remove the duplicated part in test_operator_gpu.py.

[eric-haibin-lin]

Thanks for the reply. Look forward to the update

[pengzhao-intel]

Is this PR good to merge? @TaoLv @eric-haibin-lin

[TaoLv]

@eric-haibin-lin Code is re-based. memset is needed to pass the unit tests. We can revisit the kWriteInplace check in a follow up PR if we notice any performance problem there. As the FInplaceOption is not enabled for these operators, I think it's safe for not checking it. Let me know what do you think. Thnanks!

[eric-haibin-lin]

comments addressed

[szha]

But since the oldest cuda version in CI is 9.2 so I think it's safe to remove this check and copy tests to test_operator.py. I will remove the duplicated part in test_operator_gpu.py.

@TaoLv This is not true for the CD pipeline for cu90. See this failure

[TaoLv]

@szha Any suggestion for this case? It's shared for both CPU and GPU test, can we add the cuda requirement decorator to it? Also if we're still releasing cu90 flavor, why it was removed from CI?