[Inference] add config.enable_low_precision_io api and remove rely on AnalysisConfig::Precison in trt

[yuanlehome]

PR types

New features

PR changes

APIs

Description

• 移除Paddle-TRT底层实现对上层接口AnalysisConfig::Precision的依赖，内部使用phi::DataType；
• 新增config. enable_low_precision_io接口来设置使用混合精度推理时输入输出数据的类型，此接口限制在模型结构为fp32以及开启GPU混合精度模式下使用；
  • 默认为false(即不调用此api)，即输入输出数据类型为fp32
    • 若模型为fp32，需要喂float数据，会吐出float数据（常规情况）
  • 若设置为true，需要喂float16数据，会吐出flaot16数据（可能有特殊情况，即最后一个op只支持fp32或只能运行在cpu上，此时会吐出float数据）（本PR专门支持的情况）
  • 未列出的其他情况，属于不合理情况，行为未定义。

fix bug: #54042, #54032, #54129

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[leo0519]

Hi @yuanlehome , is there any unit test for the new config keep_io_datatype?

[yuanlehome]

Hi @yuanlehome , is there any unit test for the new config keep_io_datatype?

Currently not available, I need to resolve some unitest issues on CI first.
Perhaps you can tell me what kind of usage examples you need？TRT? Naive GPU? Model precision?

[leo0519]

Perhaps you can tell me what kind of usage examples you need？TRT? Naive GPU? Model precision?

A simple network fed into Paddle Inference with the following conditions is good to check the input datatype of TRT engine.

1. enable_use_gpu is set
2. enable_tensorrt_engine is set with precision FP16
3. keep_io_datatype is set to be false

The result predictor in Paddle Inference should take an FP16 GPU tensor as input and an FP16 tensor as output.

To test the IR graph that adds cast, a program consisting of some operators unable to be converted into TRT is also required.

[yuanlehome]

Perhaps you can tell me what kind of usage examples you need？TRT? Naive GPU? Model precision?

A simple network fed into Paddle Inference with the following conditions is good to check the input datatype of TRT engine.

1. enable_use_gpu is set
2. enable_tensorrt_engine is set with precision FP16
3. keep_io_datatype is set to be false

The result predictor in Paddle Inference should take an FP16 GPU tensor as input and an FP16 tensor as output.

To test the IR graph that adds cast, a program consisting of some operators unable to be converted into TRT is also required.

Good tips. Maybe I want to wait for this PR to merge before proceeding.

[leo0519]

Hi @yuanlehome , I would like to add more information for the UT:

There are two parts of the UT

• Test FP16 IO

  1. Initialize a paddle program with a simple network (ex. fullyconnected -> batchnorm -> relu -> ... those OPs can be converted into TRT)
  2. Run Paddle Inference with GPU on and TensorRT off, and then take the output as baseline.
  3. Run Paddle Inference with GPU on, TensorRT on (TRT precision is set FP16), and keep_io_datatype False
  4. Feed FP16 input data and compare the output with baseline.

• Test auto_mixed subgraph pass

  1. Initialize a paddle program with some operators Paddle-TRT cannot convert.
  2. Run Paddle Inference with GPU on and TensorRT off and take the output as baseline.
  3. Run Paddle Inference with GPU on and TensorRT on (FP16 precision), and keep_io_datatype False
  4. Feed input data and compare the output with baseline.

If there are any questions, please let me know.

[paddle-ci-bot]

Sorry to inform you that 44f13a0's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

[XieYunshen]

LGTM for set_tests_properties(test_trt_inference_fp16_io PROPERTIES TIMEOUT 300)

[jiweibo]

LGTM

[XiaoguangHu01]

LGTM

[zhangjun]

LGTM