Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Add epsilon for tolerance level to fix flaky test random_resize_crop #15098

Merged
merged 1 commit into from
May 31, 2019
Merged

Add epsilon for tolerance level to fix flaky test random_resize_crop #15098

merged 1 commit into from
May 31, 2019

Conversation

stu1130
Copy link
Contributor

@stu1130 stu1130 commented May 29, 2019 •
edited
Loading

Description

check all the seeds listed below

seed w h
1948053179 80 107
1781948197 80 107
2015925942 110 147
869412463 155 207
1765468767 92 123

Those ration of width and height are around 0.748 and cross compared with Pytorch unit test found out that we need epsilon.

ran 10k times

# I comment all the unit tests except test_random_size_crop
MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s test_image.py
...
[DEBUG] 9984 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=985181154 to reproduce.
[DEBUG] 9985 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=364814904 to reproduce.
[DEBUG] 9986 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=670967044 to reproduce.
[DEBUG] 9987 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=323906397 to reproduce.
[DEBUG] 9988 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=918973098 to reproduce.
[DEBUG] 9989 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=770423479 to reproduce.
[DEBUG] 9990 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=480885263 to reproduce.
[DEBUG] 9991 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1533878544 to reproduce.
[DEBUG] 9992 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1124265466 to reproduce.
[DEBUG] 9993 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1208755314 to reproduce.
[DEBUG] 9994 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=454440019 to reproduce.
[DEBUG] 9995 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=647574282 to reproduce.
[DEBUG] 9996 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1521865091 to reproduce.
[DEBUG] 9997 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1304174751 to reproduce.
[DEBUG] 9998 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1949421096 to reproduce.
[DEBUG] 9999 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=732334938 to reproduce.
[DEBUG] 10000 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1699843029 to reproduce.
ok
cleanup /tmp/tmp6Lg6ez

----------------------------------------------------------------------
Ran 1 test in 192.979s

OK

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Comments

@zhreshold @abhinavs95

@stu1130 stu1130 changed the title add epsilon for tolerance level for flaky test random_resize_crop add epsilon for tolerance level to fix flaky test random_resize_crop May 29, 2019
@stu1130 stu1130 changed the title add epsilon for tolerance level to fix flaky test random_resize_crop Add epsilon for tolerance level to fix flaky test random_resize_crop May 29, 2019
@abhinavs95
Copy link
Contributor

@mxnet-label-bot add [Python, Test, Flaky]

@stu1130 stu1130 force-pushed the fix_flaky_random_size_crop branch 2 times, most recently from aba2bea to f1fc081 Compare May 30, 2019 23:33
@stu1130 stu1130 force-pushed the fix_flaky_random_size_crop branch from f1fc081 to 64836af Compare May 31, 2019 06:43
abhinavs95
abhinavs95 approved these changes May 31, 2019
View reviewed changes
Copy link
Contributor

@abhinavs95 abhinavs95 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the fix!

apeforest
apeforest reviewed May 31, 2019
View reviewed changes
tests/python/unittest/test_image.py
def test_random_size_crop(self):
# test aspect ratio within bounds
width = np.random.randint(100, 500)
height = np.random.randint(100, 500)
src = np.random.rand(height, width, 3) * 255.
ratio = (0.75, 1)
epsilon = 0.05
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we decide epsilon? is 0.05 too large here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @stu1130 for the discussion offline. The randomness is caused by sqrt of area / or * ratio. Same epsilon is used in pytorch test as well. Approve this change thereby.

apeforest
apeforest approved these changes May 31, 2019
View reviewed changes
Copy link
Contributor

@apeforest apeforest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@apeforest apeforest merged commit 6f01a6f into apache:master May 31, 2019
haohuanw pushed a commit to haohuanw/incubator-mxnet that referenced this pull request Jun 23, 2019
@stu1130
add epsilon for tolerance level (apache#15098)
9c7db27
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@abhinavs95 abhinavs95 abhinavs95 approved these changes

@apeforest apeforest apeforest approved these changes

@zhreshold zhreshold zhreshold approved these changes

Assignees
No one assigned
Labels
Flaky Python Test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@stu1130 @abhinavs95 @zhreshold @apeforest @marcoabreu