[Auto parallel] Mixed Precision FP16 Pass

[JZ-LIANG]

PR types

New features

PR changes

Others

Describe

Add Mixed Precision FP16 Pass for Auto parallel

Test on GPT-medium model with bsz = 16 x 512, V100 32G

	速度 token/s	显存 MB
Baseline	4489	27545
AMP Pass (O1)	11898	19823
FP16 Pass (O2)	12976	18668
FP16 Pass (O3)	13235	16693

Usage

        dist_strategy = fleet.DistributedStrategy()
        dist_strategy.amp =True
        white_list = ['gelu']
        black_list = ['layer_norm', 'softmax'] 
        dist_strategy.amp_configs = {
            "custom_white_list": white_list,
            "custom_black_list": black_list,
            "init_loss_scaling": 32768,
            "use_dynamic_loss_scaling": True,
            "use_pure_fp16": True, # True to enable O2 level
            "use_optimizer_fp16": True,  # True to enable O3 level
        }

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[aoyulong]

LGTM.