[AMP OP&Test] Fix scale kernel and perfect unit test #50998

201716010711 · 2023-02-28T04:23:23Z

PR types

Others

PR changes

Others

Describe

修复scale fp16和bf16的精度损失问题，精简scale fp16 单测

paddle-bot · 2023-02-28T04:23:27Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

Vvsmile · 2023-02-28T04:51:48Z

python/paddle/fluid/tests/unittests/test_scale_op.py

+class TestScaleFp16Op1(TestScaleOp):
+    def init_dtype_type(self):
+        self.dtype = np.float16
+        self.inputs = {'X': np.random.random((10, 20, 20)).astype(self.dtype)}


建议用uniform的形式替换random生成

Vvsmile · 2023-02-28T04:51:53Z

python/paddle/fluid/tests/unittests/test_scale_op.py

+    def test_check_output(self):
+        place = core.CUDAPlace(0)
+        if core.is_float16_supported(place):
+            self.check_output_with_place(place, atol=0.001, check_eager=True)


这里的atol可以不加了，里面已经默认成了1e-3

Vvsmile · 2023-02-28T05:01:11Z

python/paddle/fluid/tests/unittests/test_scale_op.py

+    def init_dtype_type(self):
+        self.dtype = np.float16
+        self.inputs = {
+            'X': np.random.uniform(-10, 10, (10, 20, 20)).astype(self.dtype)


random里面的给的参数是shape，范围是0-1的，uniform这个建议采用0，1为范围，shape与原测试保持一致

ZzSean · 2023-03-01T08:22:43Z

python/paddle/fluid/tests/unittests/test_scale_op.py

@@ -178,6 +178,22 @@ def test_check_grad(self):
        self.check_grad(['X'], 'Out', numeric_grad_delta=0.8, check_eager=True)


+class TestScaleFp16Op1(TestScaleOp):


这里上面就有FP16的测试，直接改那个就可以

ZzSean · 2023-03-02T08:25:56Z

paddle/phi/kernels/gpu/scale_kernel.cu

@@ -41,6 +41,28 @@ struct ScaleFunctor {
  }
 };

+struct ScaleFunctorFp16 {


可以重写下ScaleFunctor，增加一个模版参数，表示scale和bias的数据类型，如下
template <typename DataT， typename ParamT>
struct ScaleFunctor {}

ZzSean · 2023-03-02T11:23:36Z

改下PR Title和PR描述

ZzSean · 2023-03-02T11:26:12Z

python/paddle/fluid/tests/unittests/test_scale_op.py

@@ -149,15 +149,12 @@ def init_dtype_type(self):

    def test_check_output(self):
        place = core.CUDAPlace(0)


你这个不能都删掉，至少得运行一下self.check_output

ZzSean · 2023-03-02T11:26:28Z

python/paddle/fluid/tests/unittests/test_scale_op.py

@@ -149,15 +149,12 @@ def init_dtype_type(self):

    def test_check_output(self):
        place = core.CUDAPlace(0)


ZzSean · 2023-03-02T11:27:04Z

python/paddle/fluid/tests/unittests/test_scale_op.py

-                place, ["X"], "Out", max_relative_error=0.05, check_eager=True
-            )
+        self.check_grad_with_place(
+            place, ["X"], "Out", max_relative_error=0.001, check_eager=True


这里调的函数也不对啊，是self.check_grad

已修改，位置在#51083

ZzSean

LGTM

* Fix scale kernel for low precision, cherry pick #50998. * Fix the FP16 precision problem of add_n. (#50129) * Change squared_l2_norm to reuse ReduceKernel, and register fp16 and bf16 kernel, which is cherry pick #48315. * Cherry-pick the fix of MPTypeTrait in KP, which is implemented in #50993. * Cherry-pick the multi-precision support of AdamW for bf16, #48041. * Fix compiling error. * Cherry-pick the fix of CubTensorReduceImpl for bfloat16 in #50993. * Fix unittest. --------- Co-authored-by: liuruyan <44316842+liuruyan@users.noreply.github.com>

fix scale fp16 test

764bccb

Vvsmile reviewed Feb 28, 2023

View reviewed changes

fix scale fp16 test1

b70aa99

Vvsmile reviewed Feb 28, 2023

View reviewed changes

201716010711 added 2 commits February 28, 2023 05:24

fix scale fp16 test2

a8e3250

fix scale fp16 test3

164c867

ZzSean reviewed Mar 1, 2023

View reviewed changes

fix scale fp16 test4

e9d359e

ZzSean reviewed Mar 2, 2023

View reviewed changes

fix scale fp16 test5

8e8cd7d

ZzSean reviewed Mar 2, 2023

View reviewed changes

201716010711 changed the title ~~fix scale fp16 test~~ Fix scale kernel and perfect unit test Mar 2, 2023

201716010711 added 2 commits March 2, 2023 11:46

fix scale fp16 test6

5b2f199

fix scale fp16 test7

6be22bc

zkh2016 approved these changes Mar 6, 2023

View reviewed changes

ZzSean reviewed Mar 6, 2023

View reviewed changes

ZzSean approved these changes Mar 6, 2023

View reviewed changes

ZzSean changed the title ~~Fix scale kernel and perfect unit test~~ [AMP OP&Test] Fix scale kernel and perfect unit test Mar 6, 2023

ZzSean merged commit 02f6674 into PaddlePaddle:develop Mar 6, 2023

Xreki added a commit to Xreki/Paddle that referenced this pull request Apr 9, 2023

Fix scale kernel for low precision, cherry pick PaddlePaddle#50998.

0ac7cb0

Xreki mentioned this pull request Apr 11, 2023

Cherry pick for fix of operator precision. #52705

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AMP OP&Test] Fix scale kernel and perfect unit test #50998

[AMP OP&Test] Fix scale kernel and perfect unit test #50998

201716010711 commented Feb 28, 2023 •

edited

Loading

paddle-bot bot commented Feb 28, 2023

Vvsmile Feb 28, 2023

201716010711 Mar 2, 2023

Vvsmile Feb 28, 2023

201716010711 Mar 2, 2023

Vvsmile Feb 28, 2023

201716010711 Mar 2, 2023

ZzSean Mar 1, 2023

201716010711 Mar 2, 2023

ZzSean Mar 2, 2023

201716010711 Mar 2, 2023

ZzSean commented Mar 2, 2023 •

edited

Loading

ZzSean Mar 2, 2023

ZzSean Mar 2, 2023

ZzSean Mar 2, 2023

201716010711 Mar 7, 2023

ZzSean left a comment

		@@ -178,6 +178,22 @@ def test_check_grad(self):
		self.check_grad(['X'], 'Out', numeric_grad_delta=0.8, check_eager=True)


		class TestScaleFp16Op1(TestScaleOp):

		@@ -149,15 +149,12 @@ def init_dtype_type(self):

		def test_check_output(self):
		place = core.CUDAPlace(0)

[AMP OP&Test] Fix scale kernel and perfect unit test #50998

[AMP OP&Test] Fix scale kernel and perfect unit test #50998

Conversation

201716010711 commented Feb 28, 2023 • edited Loading

PR types

PR changes

Describe

paddle-bot bot commented Feb 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZzSean commented Mar 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZzSean left a comment

Choose a reason for hiding this comment

201716010711 commented Feb 28, 2023 •

edited

Loading

ZzSean commented Mar 2, 2023 •

edited

Loading