Benchmark/mnist

[dzhwinter]

Since this PR contains TensorFlow code, this benchmark will not be merged, the small fix has been included in our develop branch.

The comparison result is shown in #5862

[helinwang]

LGTM

[reyoung]

Do not change default random seed to 1.

[qingqing01]

Better to add testing set in the demo.

[QiJune]

What's the purpose of this PR? Does it want to compare the acc with other frameworks, like tensorflow/pytorch?

[dzhwinter]

I compare our refactorization with Paddle V1 / TensorFlow.
The reason I change the random seed into 1 --- Paddle V1 cannot set a 0 random seed, 0 means the seed selected by the system in the old config. Just for comparing the results.

[dzhwinter]

I will merge the tensorflow code into this PR, and make it more clear. Sorry for the misleading.

[QiJune]

And what's the result of this benchmark? Does this PR compare the computation batch by batch? If we initialize parameter the same, does each batch's result is the same with tensorflow? We may need to control the error against tensorflow under 1e-6 with double compile.

[emailweixu]

How much is the effect of this initialization?
I know this change make it same as v2 initialization. But perhaps the default XavierInitializer is more common choice. We should allow the user to supply initializer. In fact, it is can be supplied in param_attr. Adding an initializer here will ignore the initializer setting in param_attr and cause confusing.

In order to make the sample mnist code, it is better to change the example to set the initializer through param_attr. And if the different std does not have much effect, we don't even need to bother to change the mnist example.

[dzhwinter]

Sorry for such a late reply caused by the trip. I compared the forward cost, accuracy, loss with v2/tensorflow batch by batch. Because our default Initializer is UniformInitializer, has a
-0.02 compared with the benchmark at the first several batches. With the NormalInitalizer w e have reached the same result with Paddle V2, even at the last decimal. I think we can draw the conclusion that our implementation has got the same accuracy with the Paddle V2.

Tensorflow uses a special random number generator algorithm Philox which we don't have. We can reach the approximal same result when we fill Variable with the same random value, but there is a difference at 6 to 8 decimal place, caused by the conv2d operator implementation difference.

We should allow the user to supply initializer. In fact, it is can be supplied in param_attr. Adding an initializer here will ignore the initializer setting in param_attr and cause confusing.

We are working on make Initializer configurable. . Sure will only change the mnist example configuration in the benchmark test and set fc layer with XavierInitilizer as default.
#5819
#5805
#5760