test_operator.test_dropout has fixed seed that can mask flakiness

[szha]

The unit test in title have been using fixed seed to mask flakiness. Suggested action:

1. Evaluate whether the test is flaky without fixed seed. If not, remove seed. Else move to 2
2. If test is flaky, determine whether it's an actual uncaught edge case. If so, fix the operator. Else move to 3
3. If numerical instability is inevitable, adjust tolerance level appropriately.

[apeforest]

Thannks for filing this issue. We will investigate this Flaky test

[piyushghai]

Able to reproduce the issue with seeds 111913613, 211508467, 1329041279

Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/home/ubuntu/incubator-mxnet/tests/python/unittest/common.py", line 172, in test_new orig_test(*args, **kwargs) File "/home/ubuntu/incubator-mxnet/tests/python/unittest/test_operator.py", line 5610, in test_dropout check_dropout_ratio(0.0, shape) File "/home/ubuntu/incubator-mxnet/tests/python/unittest/test_operator.py", line 5554, in check_dropout_ratio assert exe.outputs[0].asnumpy().min() == min_value AssertionError:

[piyushghai]

This issue is also tracked in #9816 .

This test fails only for the case where ratio = 0
On checking, exe.outputs[0].asnumpy().min() gives the value 0,
while min_value is 1.

This seems to be an issue where the dropout operator (with ratio 0) is still dropping some values.

[piyushghai]

On running this on CPU, the flaky error is reproducible, as mentioned in the above comments.
On running this on GPU for 100k times, the test does not fail.

MXNET_TEST_COUNT=100000  nosetests --logging-level=DEBUG --verbose -s tests/python/gpu/test_operator_gpu.py:test_dropout
/home/ubuntu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
/home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  inspect.getargspec(func)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1058042572 to reproduce.
test_operator_gpu.test_dropout ... [DEBUG] 1 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1077219613 to reproduce.
[DEBUG] 2 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1986463962 to reproduce.
[DEBUG] 3 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1446820082 to reproduce.
[DEBUG] 4 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1300194532 to reproduce.
[DEBUG] 5 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=576630976 to reproduce.
[DEBUG] 6 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1613399106 to reproduce.
.
.
.

[DEBUG] 99996 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=92629950 to reproduce.
[DEBUG] 99997 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=104203650 to reproduce.
[DEBUG] 99998 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1609587233 to reproduce.
[DEBUG] 99999 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=194799359 to reproduce.
[DEBUG] 100000 of 100000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=675994555 to reproduce.
ok

----------------------------------------------------------------------
Ran 1 test in 7130.900s

OK

On running on GPU with the failure seeds for CPU : 111913613, 211508467, 1329041279, the test still does not fail.

MXNET_TEST_SEED=1329041279 nosetests --logging-level=DEBUG --verbose -s tests/python/gpu/test_operator_gpu.py:test_dropout
/home/ubuntu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
/home/ubuntu/anaconda3/lib/python3.6/site-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  inspect.getargspec(func)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1997611840 to reproduce.
/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py:244: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn('*** test-level seed set: all "@with_seed()" tests run deterministically ***')
[WARNING] *** test-level seed set: all "@with_seed()" tests run deterministically ***
test_operator_gpu.test_dropout ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1329041279 to reproduce.
ok

----------------------------------------------------------------------
Ran 1 test in 2.103s

OK

[piyushghai]

@sandeep-krishnamurthy Related fix for this flaky test is merged, this issue should be good to close.