Run Python OP tests in a single Python process to improve test time. #8362
Conversation
|
|
There was a problem hiding this comment.
Thanks to @panyx0718 for working over the weekend on this very urgent project!
cmake/generic.cmake
Outdated
| @@ -483,6 +483,19 @@ function(py_test TARGET_NAME) | |||
| endif() | |||
| endfunction() | |||
|
|
|||
| function(py_test_modules TARGET_NAME) | |||
There was a problem hiding this comment.
I like py_test_modules and it has the potential in many other projects. For the near future, is it the only purpose of py_test_modules to bundle operators unit tests? If so, how about we move it to fluid/tests/unittests/CMakeLists.txt?
| py_test_modules(test_conditional_block MODULES test_conditional_block) | ||
|
|
||
| # Allow to split the test so that developer can invoke a specific test. | ||
| if(WITH_FAST_BUNDLE_TEST) |
There was a problem hiding this comment.
It seems that we'd always want to run tests the fast way. How about we remove the CMake option WITH_FAST_BUNDLE_TEST by assuming it is true?
There was a problem hiding this comment.
When one test within the bundled test fails, user might want to only test that failed test instead of testing all tests in the bundle (takes a long time)?
There was a problem hiding this comment.
Good point! With the current solution, it seems that the user need to re-run cmake, like the following:
ctest # assume that a Python test failed
cmake -DWITH_FAST_BUNDLE_TEST=OFF .. # re-run cmake to swtich to the "slow" mode
make # re-build as we re-run cmake
ctest -V -R the_failed_python_testAm I correctly understanding the situation? If so, I am afraid that the users might want a quicker solution? Is it a good idea if (1) we assume WITH_FAST_BUNDLE_TEST is always true and remove it, and (2) we add a Gflag in our Python test framework that works like the -R option of ctest?
Anyway, if this PR passes, I'd like to merge it, before we find a solution for running a specified Python test in the bundle.
There was a problem hiding this comment.
The PR passes now. I can think about your proposal as a followup
panyx0718
force-pushed
the
develop
branch
3 times, most recently
from
04af0dc to
972a7c9
Compare
|
Thanks, PTAL |
panyx0718
force-pushed
the
develop
branch
10 times, most recently
from
d34e5ec to
7896e97
Compare
|
It seems "continuous-integration/travis-ci" is not working? |
|
The travis-ci seems normally, maybe you can push a commit again. |
|
Thanks, I resolved that conflicts and it seems to work now. |
panyx0718
force-pushed
the
develop
branch
5 times, most recently
from
6bba41a to
bb00660
Compare
Currently, our tests run with 2 GPUs, the init time is absurdly long: about 4s for each process. Currently, we run each OP test on different processes. This PR: 1. create cmake function py_test_modules which will generate the Makefile that runs a list of Python unittest module in a single Python process. 2. move all "python unittest compatible" (e.g., used the unittest package, not just a regular python file). from fluid/tests to fluid/tests/unittests. 3. cmake now will run all OP tests in fluid/tests/unittests in a single process, except the time-consuming tests, they are separated into different processes to utilize parallelism. Please make sure to use the unittest package if you put the python test file in fluid/tests/unittests 4. remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used to disable unittest, we can not do it when running all tests in a single process since it will terminate the process without running the other tests. Instead, the test is disabled in fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled item. Please disable the unittest from fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the Python file, for all Python file in fluid/tests/unittests/. 5. add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit tests in separate process so that they can be tested individually.
Currently, our tests run with 2 GPUs, the init time is absurdly long:
about 4s for each process. Currently, we run each OP test on
different processes. This PR:
create cmake function py_test_modules which will generate the
Makefile that runs a list of Python unittest module in a single Python
process.
move all "python unittest compatible" (e.g., used the unittest
package, not just a regular python file). from fluid/tests to
fluid/tests/unittests.
cmake now will run all OP tests in fluid/tests/unittests in a
single process, except the time-consuming tests, they are separated
into different processes to utilize parallelism. Please make sure to
use the unittest package if you put the python test file in
fluid/tests/unittests
remove all exit(0) from fluid/tests/unittests/*.py, exit(0) is used
to disable unittest, we can not do it when running all tests in a
single process since it will terminate the process without running the
other tests. Instead, the test is disabled in
fluid/tests/unittests/CMakeLists.txt. FIXME is added for each disabled
item. Please disable the unittest from
fluid/tests/unittests/CMakeLists.txt, instead of adding exit(0) to the
Python file, for all Python file in fluid/tests/unittests/.
add an option WITH_FAST_BUNDLE_TEST. When OFF, will run the unit
tests in separate process so that they can be tested individually.