Enhance the unit test framework to explicitly test whether the operator correctly handles gradients for multiple inputs.

[qingqing01]

Fix #3797

If all input gradients of forwarding operator do not need to calculate, the backward operator will be a NOP operator: /~https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/framework/backward.cc#L75

So, there is no need to handle this case for the backward op with only one output.

[reyoung]

one of more of the gradient of the input is --> some input gradients are

[reyoung]

Awesome work for making mul/rowwise_add can ignore some IG.

However, I think the gradient unit-test framework should not be changed for the following reason.

If a user wants to check a mul operator gradient without some IGs, he can invoke check_grad many times, like check_grad("mul", input_to_check=["X"], ignore=["Y"]), check_grad("mul", input_to_check["Y"], ignore=["X"]).

• The disadvantage for invoking check_grad many times is that the numeric gradient will be calculated many times. It could be slow, but unit tests are not used by users, so it is OK.
• To make users invoke check_grad by themselves can handle if some IGs are not able to be ignored. For example, OpA has two inputs, 'X' and 'Y' and gradient 'Y' should always be calculated, and it YG should never be nullptr.
• To make users invoke check_grad can make error log clearer. The error could be check mul gradient error when calculating X and ignore Y.

[reyoung]

Please review qingqing01#1

[qingqing01]

@reyoung Thanks！I also thought it was not good to change the check_grad code at first. I merge your patch.

[reyoung]

LGTM.

[emailweixu]

@qingqing01 The framework should automatically test cases with certain input gradient ignored without explicitly writing code here. We should change GradientChecker.check_grad() to do this.

[qingqing01]

@emailweixu Thanks for your comments. Ok, in the first comments: 4470332, the check_grad is automatically testing. I'll change it again :).

[reyoung]

The framework should automatically test cases with certain input gradient ignored without explicitly writing code here. We should change GradientChecker.check_grad() to do this.

• To make users invoke check_grad by themselves can handle if some IGs are not able to be ignored. For example, OpA has two inputs, 'X' and 'Y' and gradient 'Y' should always be calculated, and it YG should never be nullptr.
  • Also, check_grad is not only used to check single operator's gradient but also the gradient inside a whole neural network. It is hard to automatically testing a whole network.
• To make users invoke check_grad can make error log clearer. The error would be test_mul_grad_ingore_x error.
• To make user-interface easily, there maybe a parameter in check_grad, like test_no_input_grad. If test_no_input_grad=True, multiple check_grads will be invoked.