ADTM KNL build on mutrino to use OMP_NUM_THREADS=2 #3278

fryeguy52 · 2018-08-10T17:34:37Z

Description

Change atdm KNL build on mutrino to use OMP_NUM_THREADS=2 previously it was set to 16. This change in local testing of the panzer test suite caused the tests to run in about 1/2 of the time.

Motivation and Context

The panzer tests were being run in about 1/2 the time in the Empire KNL build on mutrino. This is to get the ATDM build to finish in a comparable time

How Has This Been Tested?

I tested this locally by runnig tests with the two different values for OMP_NUM_THREADS

with OMP_NUM_THREADS=2 Total Test time is 132.80 sec

ctest -j8 -R PanzerAdaptersSTK_CurlLaplacianExample
Test project /lscratch1/jfrye/SRC_AND_BUILD/BUILD
    Start 133: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4
    Start 132: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3
1/5 Test #132: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3 ...   Passed   44.96 sec
    Start 131: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2
2/5 Test #133: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 ...   Passed   77.43 sec
    Start 130: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1
3/5 Test #131: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2 ...   Passed   68.40 sec
    Start 129: PanzerAdaptersSTK_CurlLaplacianExample
4/5 Test #130: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1 ...   Passed   41.77 sec
5/5 Test #129: PanzerAdaptersSTK_CurlLaplacianExample .........................   Passed   16.15 sec

100% tests passed, 0 tests failed out of 5

Subproject Time Summary:
Panzer    = 994.88 sec*proc (5 tests)

Total Test time (real) = 132.80 sec

env | sort|grep -i omp_
OMP_NUM_THREADS=2

with OMP_NUM_THREADS=16 total test time is 407.84 sec

ctest -j8 -R PanzerAdaptersSTK_CurlLaplacianExample
Test project /lscratch1/jfrye/SRC_AND_BUILD/BUILD
    Start 133: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4
    Start 132: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3
1/5 Test #133: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 ...   Passed  269.91 sec
    Start 131: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2
2/5 Test #132: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3 ...   Passed  315.79 sec
    Start 130: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1
3/5 Test #131: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2 ...   Passed  104.72 sec
    Start 129: PanzerAdaptersSTK_CurlLaplacianExample
4/5 Test #130: PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1 ...   Passed   67.46 sec
5/5 Test #129: PanzerAdaptersSTK_CurlLaplacianExample .........................   Passed   30.81 sec

100% tests passed, 0 tests failed out of 5

Subproject Time Summary:
Panzer    = 3154.75 sec*proc (5 tests)

Total Test time (real) = 407.84 sec

env | sort|grep -i omp_
OMP_NUM_THREADS=16

updade mutrino envronment script to use OMP_NUM_THREADS=2 instead of OMP_NUM_THREADS=16. In local testing this makes the test suite for panzer run about twice as fast

trilinos-autotester · 2018-08-10T17:43:36Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

trilinos-autotester · 2018-08-10T17:45:40Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 1330
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 1024
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 574
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Using Repos:

Repo: TRILINOS (fryeguy52/Trilinos)

Pull Request Author: fryeguy52

trilinos-autotester · 2018-08-10T18:41:38Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 1330
Status: FAILED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 1024
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 574
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

CDash Test Results for PR# 3278.

bartlettroscoe · 2018-08-10T21:36:07Z

FYI: For details on the above PR testing failure see #3276 (comment).

bartlettroscoe

Looks good.

bartlettroscoe · 2018-08-10T21:39:19Z

cmake/std/atdm/mutrino/environment.sh

@@ -51,7 +51,7 @@ if [ "$ATDM_CONFIG_COMPILER" == "INTEL" ] && [ "$ATDM_CONFIG_KOKKOS_ARCH" == "HS
 elif [ "$ATDM_CONFIG_COMPILER" == "INTEL" ] && [ "$ATDM_CONFIG_KOKKOS_ARCH" == "KNL"  ]; then
    module use /projects/EMPIRE/mutrino/tpls/knl/modulefiles
    export SLURM_TASKS_PER_NODE=16
-    export OMP_NUM_THREADS=16


Wow, I can't believed I messed that when I was looking at this. Going for 16 down to 2 threads per process should make a huge difference.

trilinos-autotester · 2018-08-11T00:27:17Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

trilinos-autotester · 2018-08-11T00:27:40Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 1336
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 1030
Status: STARTED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 579
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Using Repos:

Repo: TRILINOS (fryeguy52/Trilinos)

Pull Request Author: fryeguy52

trilinos-autotester · 2018-08-11T00:31:29Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Build Num: 1336
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.9.3
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Build Num: 1030
Status: PASSED

Jenkins Parameters

Parameter Name	Value
COMPILER_MODULE	sems-gcc/4.8.4
JENKINS_BUILD_TYPE	Release
JENKINS_COMM_TYPE	MPI
JENKINS_DO_COMPLEX	OFF
JENKINS_JOB_TYPE	Experimental
MPI_MODULE	sems-openmpi/1.8.7
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Build Num: 579
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PULLREQUESTNUM	3278
TEST_REPO_ALIAS	TRILINOS
TRILINOS_SOURCE_BRANCH	atdm-KNL-mutrino
TRILINOS_SOURCE_REPO	/~https://github.com/fryeguy52/Trilinos
TRILINOS_SOURCE_SHA	`6c02877`
TRILINOS_TARGET_BRANCH	develop
TRILINOS_TARGET_REPO	/~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA	`e91cdde`

CDash Test Results for PR# 3278.

trilinos-autotester · 2018-08-11T00:31:49Z

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ bartlettroscoe mhoemmen ]!

trilinos-autotester · 2018-08-11T00:31:58Z

Status Flag 'Pull Request AutoTester' - AutoMerge IS ENABLED, but the Label AT: AUTOMERGE is not set. Either set Label AT: AUTOMERGE or manually merge the PR...

updade mutrino envronment script to use OMP_NUM_THREADS=2 instead of OMP_NUM_THREADS=16. In local testing this makes the test suite for panzer run about twice as fast

ADTM KNL build on mutrino to use OMP_NUM_THREADS=2

6c02877

updade mutrino envronment script to use OMP_NUM_THREADS=2 instead of OMP_NUM_THREADS=16. In local testing this makes the test suite for panzer run about twice as fast

fryeguy52 added client: ATDM Any issue primarily impacting the ATDM project ATDM Config Issues that are specific to the ATDM configuration settings labels Aug 10, 2018

fryeguy52 added this to the Keep promoted "ATDM" builds of Trilinos clean milestone Aug 10, 2018

fryeguy52 requested a review from bartlettroscoe August 10, 2018 17:34

bartlettroscoe added the stage: in progress Work on the issue has started label Aug 10, 2018

mhoemmen approved these changes Aug 10, 2018

View reviewed changes

mhoemmen added the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Aug 10, 2018

bartlettroscoe mentioned this pull request Aug 10, 2018

Trilinos auto PR tester stability issues #3276

Closed

bartlettroscoe approved these changes Aug 10, 2018

View reviewed changes

trilinos-autotester removed the AT: RETEST Causes the PR autotester to run a new round of PR tests on the next iteration label Aug 11, 2018

bartlettroscoe merged commit f2b6ef2 into trilinos:develop Aug 11, 2018

bartlettroscoe removed the stage: in progress Work on the issue has started label Aug 11, 2018

trilinos-autotester mentioned this pull request Aug 10, 2021

Belos: Fix Output Style Crash, Issue #9522 #9539

Merged

trilinos-autotester mentioned this pull request Oct 8, 2021

MueLu: refactor residual printout in Hierarchy::Iterate() #9793

Merged

ADTM KNL build on mutrino to use OMP_NUM_THREADS=2 #3278

ADTM KNL build on mutrino to use OMP_NUM_THREADS=2 #3278

Conversation

fryeguy52 commented Aug 10, 2018

Description

Motivation and Context

How Has This Been Tested?

trilinos-autotester commented Aug 10, 2018

trilinos-autotester commented Aug 10, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Using Repos:

trilinos-autotester commented Aug 10, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

bartlettroscoe commented Aug 10, 2018

bartlettroscoe left a comment

Choose a reason for hiding this comment

bartlettroscoe Aug 10, 2018

Choose a reason for hiding this comment

trilinos-autotester commented Aug 11, 2018

trilinos-autotester commented Aug 11, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

Using Repos:

trilinos-autotester commented Aug 11, 2018

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

Jenkins Parameters

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

Jenkins Parameters

trilinos-autotester commented Aug 11, 2018

trilinos-autotester commented Aug 11, 2018