Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'release-debug' build type (#3633) and change some 'waterman' builds (#3336) #3659

Merged

Conversation

bartlettroscoe
Copy link
Member

CC: @fryeguy52

Description

This will allow many of the ATDM Trilinos 'debug' builds to be switched to
'release-debug' builds and help to avoid a bunch of timeouts that we are
dealing with.

Switch some 'waterman' 'debug' builds to new 'release-debug' builds (#3633)

I kept the existing cuda-9.2-debug-Power9-Volta70 build since there are
currently not any timing out tests in that build and I figured that the CUDA
build was most likey the one a developer would want to run with a debugger.
But I created a new cuda-9.2-release-debug-Power9-Volta70 build so that we can
avoid having to disable slow Kokkos, KokkosKernels, and other tests that run
super slow with -O0.

I changed the build gnu-debug-openmp-Power9-Volta70 to a
gnu-release-debug-openmp-Power9-Volta70 build since I don't think it is as
important to run this build with a debugger and the full 'debug' build and
this build currently has some timing-out tests for Kokkos and KokkosKernals as
described in #3336. (The new gnu-release-debug-openmp-Power9-Volta70 build
has not have any timeouts.) If the APP teams tell us they want a full
gnu-debug-openmp-Power9-Volta70 build, then we will add one back and deal with
the timeouts.

NOTE: By having both 'debug' and 'release-debug' builds, we can be free to
disable some slow tests in the full 'debug' build and not loose much runtime
debug checking since these tests will be running in the 'release-debug' build
(with runtime debug checking enabled). So going forward, if a test times-out
in the 'debug' build but not the 'release-debug' build, then we will just
disable it in the 'debug' build and move on.

I also renamed the 'cuda' builds to 'cuda-9.2' builds since that is what they
are and that matches the Jenkins drive names.

Motivation and Context

We want to avoid timeouts like #3336 recently and going forward (see #3633).

How Has This Been Tested?

I tested this on on 'waterman' with:

$ bsub -x -Is -n 20 \
  ./checkin-test-atdm.sh all --enable-packages=Kokkos,KokkosKernels --local-do-all

and it returned:

PASSED (NOT READY TO PUSH): Trilinos: waterman3

Wed Oct 17 12:41:58 MDT 2018

Enabled Packages: Kokkos, KokkosKernels

Build test results:
-------------------
0) MPI_RELEASE_DEBUG_SHARED_PT_OPENMP => Test case MPI_RELEASE_DEBUG_SHARED_PT_OPENMP was not run! => Does not affect push readiness! (-1.00 min)
1) gnu-release-debug-openmp-Power9-Volta70 => passed: passed=35,notpassed=0 (3.52 min)
2) gnu-opt-openmp-Power9-Volta70 => passed: passed=35,notpassed=0 (3.34 min)
3) cuda-9.2-release-debug-Power9-Volta70 => passed: passed=35,notpassed=0 (11.84 min)
4) cuda-9.2-debug-Power9-Volta70 => passed: passed=35,notpassed=0 (9.35 min)
5) cuda-9.2-opt-Power9-Volta70 => passed: passed=35,notpassed=0 (3.69 min)

A current successful pull does *not* exist => Not ready for final push!

Explanation: In order to safely push, the local working directory needs
to be up-to-date with the global repo or a full integration has not been
performed!


REQUESTED ACTIONS: PASSED

NOTE: The timing-out tests KokkosContainers_UnitTest_Serial_MPI_1 and KokkosKernels_graph_serial_MPI_1 for the updated build gnu-release-debug-openmp-Power9-Volta70 (which used to be the full debug build gnu-debug-openmp-Power9-Volta70) reported in #3336 are gone! For the build gnu-release-debug-openmp-Power9-Volta70 the ctest output showed:

30/35 Test #34: KokkosKernels_graph_serial_MPI_1 .................   Passed   48.19 sec
...
35/35 Test #25: KokkosContainers_UnitTest_Serial_MPI_1 ...........   Passed  105.93 sec

So merging this branch will fix #3336.

Checklist

  • My commit messages mention the appropriate GitHub issue numbers.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

This will allow many of the ATDM Trilinos 'debug' builds to be switched to
'release-debug' builds and help to avoid a bunch of timeouts that we are
dealing with.
…rilinos#3633)

I kept the existing cuda-9.2-debug-Power9-Volta70 build since there are
currently not any timing out tests in that build and I figured that the CUDA
build was most likey the one a developer would want to run with a debugger.
But I created a new cuda-9.2-release-debug-Power9-Volta70 build so that we can
avoid having to disable slow Kokkos, KokkosKernels, and other tests that run
super slow with -O0.

I changed the build gnu-debug-openmp-Power9-Volta70 to a
gnu-release-debug-openmp-Power9-Volta70 build since I don't think it is as
important to run this build with a debugger and the full 'debug' build and
this build currently has some timing-out tests for Kokkos and KokkosKernals as
described in trilinos#3336.  (The new gnu-release-debug-openmp-Power9-Volta70 build
has not have any timeouts.)  If the APP teams tell us they want a full
gnu-debug-openmp-Power9-Volta70 build, then we will add one back and deal with
the timeouts.

NOTE: By having both 'debug' and 'release-debug' builds, we can be free to
disable some slow tests in the full 'debug' build and not loose much runtime
debug checking since these tests will be running in the 'release-debug' build
(with runtime debug checking enabled).  So going forward, if a test times-out
in the 'debug' build but not the 'release-debug' build, then we will just
disable it in the 'debug' build and move on.

I also renamed the 'cuda' builds to 'cuda-9.2' builds since that is what they
are and that matches the Jenkins drive names.
@bartlettroscoe bartlettroscoe added pkg: Kokkos pkg: KokkosKernels client: ATDM Any issue primarily impacting the ATDM project ATDM Config Issues that are specific to the ATDM configuration settings labels Oct 17, 2018
@bartlettroscoe bartlettroscoe added stage: in progress Work on the issue has started type: enhancement Issue is an enhancement, not a bug labels Oct 17, 2018
@bartlettroscoe
Copy link
Member Author

@mhoemmen or @ibaned, one of you two around to approve this PR? This will fix some Kokkos and KokkosKernel tset timeouts :-)

@bartlettroscoe bartlettroscoe added the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Oct 17, 2018
@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

  • Build Num: 1407
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3659
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3633-atdm-release-debug
TRILINOS_SOURCE_REPO /~https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 9c42bf4
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO /~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 8d56e74

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

  • Build Num: 1970
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.9.3
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3659
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3633-atdm-release-debug
TRILINOS_SOURCE_REPO /~https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 9c42bf4
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO /~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 8d56e74

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

  • Build Num: 1632
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.8.4
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3659
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3633-atdm-release-debug
TRILINOS_SOURCE_REPO /~https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 9c42bf4
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO /~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 8d56e74

Using Repos:

Repo: TRILINOS (bartlettroscoe/Trilinos)
  • Branch: 3633-atdm-release-debug
  • SHA: 9c42bf4
  • Mode: TEST_REPO

Pull Request Author: bartlettroscoe

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: Trilinos_pullrequest_intel_17.0.1

  • Build Num: 1407
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PULLREQUESTNUM 3659
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3633-atdm-release-debug
TRILINOS_SOURCE_REPO /~https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 9c42bf4
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO /~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 8d56e74

Build Information

Test Name: Trilinos_pullrequest_gcc_4.9.3

  • Build Num: 1970
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.9.3
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3659
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3633-atdm-release-debug
TRILINOS_SOURCE_REPO /~https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 9c42bf4
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO /~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 8d56e74

Build Information

Test Name: Trilinos_pullrequest_gcc_4.8.4

  • Build Num: 1632
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
COMPILER_MODULE sems-gcc/4.8.4
JENKINS_BUILD_TYPE Release
JENKINS_COMM_TYPE MPI
JENKINS_DO_COMPLEX OFF
JENKINS_JOB_TYPE Experimental
MPI_MODULE sems-openmpi/1.8.7
PULLREQUESTNUM 3659
TEST_REPO_ALIAS TRILINOS
TRILINOS_SOURCE_BRANCH 3633-atdm-release-debug
TRILINOS_SOURCE_REPO /~https://github.com/bartlettroscoe/Trilinos
TRILINOS_SOURCE_SHA 9c42bf4
TRILINOS_TARGET_BRANCH develop
TRILINOS_TARGET_REPO /~https://github.com/trilinos/Trilinos
TRILINOS_TARGET_SHA 8d56e74


CDash Test Results for PR# 3659.

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO REVIEWS HAVE BEEN PERFORMED ON THIS PULL REQUEST!

@trilinos-autotester
Copy link
Contributor

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ ibaned ]!

@trilinos-autotester
Copy link
Contributor

Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged

@trilinos-autotester trilinos-autotester merged commit 6314c5c into trilinos:develop Oct 18, 2018
@trilinos-autotester
Copy link
Contributor

Merge on Pull Request# 3659: IS A SUCCESS - Pull Request successfully merged

@trilinos-autotester trilinos-autotester removed the AT: AUTOMERGE Causes the PR autotester to automatically merge the PR branch once approvals are completed label Oct 18, 2018
@bartlettroscoe bartlettroscoe removed the stage: in progress Work on the issue has started label Oct 18, 2018
@bartlettroscoe
Copy link
Member Author

Thanks @ibaned !

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this pull request Oct 18, 2018
…terman' cuda-9.2-debug build (trilinos#3336)

Now that this test is running and passing in the new build
Trilinos-atdm-waterman-cuda-9.2-release-debug (see trilinos#3659 and trilinos#3633), it is
fine to disable this in this full -O3 build.

 # Please
enter the commit message for your changes. Lines starting # with '#' will be
ignored, and an empty message aborts the commit.  # On branch
3336-waterman-disable-kokkoscontainers-test # Changes to be committed: #
modified:
cmake/std/atdm/waterman/tweaks/CUDA-9.2-DEBUG-CUDA-POWER9-VOLTA70.cmake #
bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this pull request Oct 18, 2018
…terman' cuda-9.2-debug build (trilinos#3336)

Now that this test is running and passing in the new build
Trilinos-atdm-waterman-cuda-9.2-release-debug (see trilinos#3659 and trilinos#3633), it is
fine to disable this in this full -O3 build.
@bartlettroscoe bartlettroscoe added ATDM DevOps Issues that will be worked by the Coordinated ATDM DevOps teams and removed ATDM DevOps Issues that will be worked by the Coordinated ATDM DevOps teams labels Oct 27, 2018
tjfulle pushed a commit to tjfulle/Trilinos that referenced this pull request Dec 6, 2018
…terman' cuda-9.2-debug build (trilinos#3336)

Now that this test is running and passing in the new build
Trilinos-atdm-waterman-cuda-9.2-release-debug (see trilinos#3659 and trilinos#3633), it is
fine to disable this in this full -O3 build.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ATDM Config Issues that are specific to the ATDM configuration settings ATDM DevOps Issues that will be worked by the Coordinated ATDM DevOps teams client: ATDM Any issue primarily impacting the ATDM project pkg: Kokkos pkg: KokkosKernels type: enhancement Issue is an enhancement, not a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants