[Dygraph]Add dygraph sharding stage3

[Baibaifan]

PR types

New features

PR changes

Others

Describe

Add dygraph sharding stage3

import paddle
from paddle.distributed.fleet.meta_parallel.sharding.sharding_stage3 import ShardingStage3

fleet.init(is_collective=True)
group = paddle.distributed.new_group([0, 1])

# wrap model
model = model_class(...)
model = ShardingStage3(model, optimizer=optimizer, group=group)

# use optimizer as normal
img, label = data
label.stop_gradient = True
img.stop_gradient = True
out = model(img)

loss = paddle.nn.functional.cross_entropy(input=out, label=label)
loss.backward()
optimizer.step()
optimizer.clear_grad()
# Get all parameter from parameter slice
model.get_all_parameters()

stage3 and DP fp16 O2 GPT 117M

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[paddle-bot-old]

Sorry to inform you that 73f82b7's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

[XieYunshen]

LGTM for set_tests_properties(test_dygraph_sharding_stage3 PROPERTIES TIMEOUT 120)

[ForFishes]

LGTM