Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Scala arange Flaky test #14402

Closed
lanking520 opened this issue Mar 12, 2019 · 5 comments · Fixed by #14417
Closed

Scala arange Flaky test #14402

lanking520 opened this issue Mar 12, 2019 · 5 comments · Fixed by #14417

Comments

@lanking520
Copy link
Member

Currently, I received several reports from contributors about the failed test they got from CI:

�[31m- arange *** FAILED ***�[0m
�[31m  NaN was not less than or equal to 1.0E-4 (NDArraySuite.scala:353)�[0m

Some problems appeared in the CI:

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-14365/7/pipeline/281

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-14359/7/pipeline

Thanks @haojin2 @yuxihu reporting these issues

@mxnet-label-bot
Copy link
Contributor

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Scala, Test, Flaky

@andrewfayres
Copy link
Contributor

Likely related to #13816. I'm trying to reproduce it locally.

@andrewfayres
Copy link
Contributor

Reproduces locally by increasing the number of iterations to 10k. Both of the tests that were added will intermittently fail. Looks like start isn't being passed to NDArray.arrange but it's a required parameter.

Tagging the author of the PR @kiendang for his input on this.

@kiendang
Copy link
Contributor

Sorry. I'm taking a look at this.

Looks like start isn't being passed to NDArray.arrange but it's a required parameter.

The value of stop was passed to start in both tests. Those two tests were added to test situations where no value is passed to stop (stop = None). The intended behavior would be what happens when start = 0 and stop = Some(<value passed to start>) as whatever was , e.g NDArray.arange(5) should be the same as NDArray.arange(0, Some(5)).
/~https://github.com/apache/incubator-mxnet/blob/30b1cbc1d862f8c5c1f91d0a9f048aef5771f9de/src/operator/tensor/init_op.h#L202-L205

@kiendang
Copy link
Contributor

Found the issue. The reason is if the two compared arrays are both like [0f] and [0f] then diff and norm will be near 0 resulting in division by zero.
/~https://github.com/apache/incubator-mxnet/blob/6aa8c27a69cafc65eb97c4e077cb399462b617e2/scala-package/core/src/test/scala/org/apache/mxnet/CheckUtils.scala#L27-L32
Solution is if diff is near 0f, i.e < Float.MinPositiveValue just return it instead of dividing by norm

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants
@kiendang @lanking520 @andrewfayres @mxnet-label-bot and others