CI JJOB Tests using CMake #3214

TerrenceMcGuinness-NOAA · 2025-01-08T18:57:38Z

Description

Adding CI tests at the JJOB level using CMake/ctest wrappers:
These individual JJOB tests have four distinct phases:

Setup: Creates a EXPDIR/COMROOT just for the individual JJOB
Stage: Moves the specific files into the COMROOT that are needed to run the JJOB specified in ${HOMEgfs}/ci/ctest/cases/{CASE}_{JJOB}.yaml
Execute: Run the JJOB in batch (batch card extracted from XML via Rocoto)
Validate: Check the outputs also specified in the above yaml configure file (currently stubbed)

Resolves #3204

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? YES
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Ran with CMake/ctest:

mterry (orion-login-2) ctest (ctest_jjobs_framework) $ mkdir build; cd build

mterry (orion-login-2) build (ctest_jjobs_framework) $ cmake ../.. -DICSDIR_ROOT=/work/noaa/global/glopara/data/ICSDIR -DHPC_ACCOUNT=nems -DSTAGED_TESTS_DIR=/work/noaa/stmp/GFS_CI_ROOT/ORION/STAGED_TESTS_DIR
-- gw: global-workflow baselines will be used from: '/work2/noaa/global/mterry/global-workflow_forked'
-- gw: global-workflow tests will be run at: '/work2/noaa/global/mterry/global-workflow_forked/ctests/build/ctests/RUNTESTS'
-- gw: global-workflow tests will use the allocation: 'nems'
-- gw: global-workflow tests will use ICSDIR_ROOT: '/work/noaa/global/glopara/data/ICSDIR'
-- gw: global-workflow tests will use staged data from:  '/work/noaa/stmp/GFS_CI_ROOT/ORION/STAGED_TESTS_DIR'
-- Build files have been written to: /work2/noaa/global/mterry/global-workflow_forked/ctests/build

mterry (orion-login-2) build (ctest_jjobs_framework) $ ctest -N
Test project /work2/noaa/global/mterry/global-workflow_forked/ctests/build
  Test #1: test_C48_ATM_gfs_fcst_seg0_setup
  Test #2: test_C48_ATM_gfs_fcst_seg0_stage
  Test #3: test_C48_ATM_gfs_fcst_seg0_execute
  Test #4: test_C48_ATM_gfs_fcst_seg0_validate

Total Tests: 4
mterry (orion-login-2) build (ctest_jjobs_framework) $ ctest -L C48_ATM
Test project /work2/noaa/global/mterry/global-workflow_forked/ctests/build
    Start 1: test_C48_ATM_gfs_fcst_seg0_setup
1/4 Test #1: test_C48_ATM_gfs_fcst_seg0_setup ......   Passed    3.25 sec
    Start 2: test_C48_ATM_gfs_fcst_seg0_stage
2/4 Test #2: test_C48_ATM_gfs_fcst_seg0_stage ......   Passed    1.90 sec
    Start 3: test_C48_ATM_gfs_fcst_seg0_execute
3/4 Test #3: test_C48_ATM_gfs_fcst_seg0_execute ....   Passed  2942.51 sec
    Start 4: test_C48_ATM_gfs_fcst_seg0_validate
4/4 Test #4: test_C48_ATM_gfs_fcst_seg0_validate ...   Passed    0.02 sec

100% tests passed, 0 tests failed out of 4

Label Time Summary:
C48_ATM          = 2947.68 sec*proc (4 tests)
gfs_fcst_seg0    = 2947.68 sec*proc (4 tests)

Total Test time (real) = 2947.68 sec

Checklist

update last bit

fix misspelled work with create

fixed a grammar in README.md

path to source as HOMEgfs

WalterKolczynski-NOAA · 2025-01-14T19:44:34Z

Execute test is failing when I try

13:42:21 <ctest_jjobs_framework> build/>ctest --output-on-failure
Test project /work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build
    Start 1: test_C48_ATM_gfs_fcst_seg0_setup
1/4 Test #1: test_C48_ATM_gfs_fcst_seg0_setup ......   Passed    2.05 sec
    Start 2: test_C48_ATM_gfs_fcst_seg0_stage
2/4 Test #2: test_C48_ATM_gfs_fcst_seg0_stage ......   Passed    1.82 sec
    Start 3: test_C48_ATM_gfs_fcst_seg0_execute
3/4 Test #3: test_C48_ATM_gfs_fcst_seg0_execute ....***Failed    0.01 sec
+ TEST_NAME=C48_ATM_gfs_fcst_seg0
+ JOB=gfs_fcst_seg0
+ idate=2021032312
+ rocotoboot_dryrun=/work2/noaa/global/mterry/rocoto_dryrun/bin/rocotoboot
+ CASEDIR=/work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build/ctests/RUNTESTS/EXPDIR/C48_ATM_gfs_fcst_seg0
+ cd /work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build/ctests/RUNTESTS/EXPDIR/C48_ATM_gfs_fcst_seg0
/work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build/ctests/scripts/execute.sh: line 13: cd: /work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build/ctests/RUNTESTS/EXPDIR/C48_ATM_gfs_fcst_seg0: No such file or directory

    Start 4: test_C48_ATM_gfs_fcst_seg0_validate
4/4 Test #4: test_C48_ATM_gfs_fcst_seg0_validate ...   Passed    0.01 sec

75% tests passed, 1 tests failed out of 4

Label Time Summary:
C48_ATM          =   3.89 sec*proc (4 tests)
gfs_fcst_seg0    =   3.89 sec*proc (4 tests)

Total Test time (real) =   3.90 sec

The following tests FAILED:
          3 - test_C48_ATM_gfs_fcst_seg0_execute (Failed)
Errors while running CTest

aerorahul · 2025-01-14T20:30:31Z

ctests/CMakeLists.txt

+# Set HPC_ACCOUNT
+set_from_env_or_default(HPC_ACCOUNT HPC_ACCOUNT " ")
+if (NOT DEFINED HPC_ACCOUNT)
+  message(FATAL_ERROR "HPC_ACCOUNT must be set")


FATAL_ERROR will stop cmake processing (I think).
The return() after this line will not allow the cmake-ification of the project.

ctests/CMakeLists.txt

modulefiles/module_gwsetup.orion.lua

ctests/scripts/validate.sh

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

… needed env vars set

…tput_files

WalterKolczynski-NOAA · 2025-01-16T09:47:56Z

I'm still getting the same error when I try to run on Hercules:

03:46:38 <ctest_jjobs_framework> pr_3214/>cd ctests/
03:46:44 <ctest_jjobs_framework> ctests/>mkdir build
03:46:47 <ctest_jjobs_framework> ctests/>cd build
03:47:04 <ctest_jjobs_framework> build/>cmake ../.. -DICSDIR_ROOT=/work/noaa/global/glopara/data/ICSDIR -DHPC_ACCOUNT=nems -DSTAGED_TESTS_DIR=/work/noaa/stmp/GFS_CI_ROOT/ORION/STAGED_TESTS_DIR
-- The C compiler identification is GNU 11.3.1
-- The CXX compiler identification is GNU 11.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting build type to 'Release' as none was specified.
-- gw: global-workflow baselines will be used from: '/work2/noaa/global/wkolczyn/save/global-workflow/pr_3214'
-- gw: global-workflow tests will be run at: '/work2/noaa/global/wkolczyn/noscrub/global-workflow'
-- gw: global-workflow tests will use the allocation: 'nems'
-- gw: global-workflow tests will use ICSDIR_ROOT: '/work/noaa/global/glopara/data/ICSDIR'
-- gw: global-workflow tests will use staged data from:  '/work/noaa/stmp/GFS_CI_ROOT/ORION/STAGED_TESTS_DIR'
-- Configuring done
-- Generating done
-- Build files have been written to: /work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build
03:47:10 <ctest_jjobs_framework> build/>ctest --output-on-failure
Test project /work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build
    Start 1: test_C48_ATM_gfs_fcst_seg0_setup
1/4 Test #1: test_C48_ATM_gfs_fcst_seg0_setup ......   Passed    4.88 sec
    Start 2: test_C48_ATM_gfs_fcst_seg0_stage
2/4 Test #2: test_C48_ATM_gfs_fcst_seg0_stage ......   Passed    2.03 sec
    Start 3: test_C48_ATM_gfs_fcst_seg0_execute
3/4 Test #3: test_C48_ATM_gfs_fcst_seg0_execute ....***Failed    0.02 sec
+ TEST_NAME=C48_ATM_gfs_fcst_seg0
+ JOB=gfs_fcst_seg0
+ idate=2021032312
+ rocotoboot_dryrun=/work2/noaa/global/mterry/rocoto_dryrun/bin/rocotoboot
+ CASEDIR=/work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build/ctests/RUNTESTS/EXPDIR/C48_ATM_gfs_fcst_seg0
+ cd /work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build/ctests/RUNTESTS/EXPDIR/C48_ATM_gfs_fcst_seg0
/work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build/ctests/scripts/execute.sh: line 13: cd: /work2/noaa/global/wkolczyn/save/global-workflow/pr_3214/ctests/build/ctests/RUNTESTS/EXPDIR/C48_ATM_gfs_fcst_seg0: No such file or directory

    Start 4: test_C48_ATM_gfs_fcst_seg0_validate
4/4 Test #4: test_C48_ATM_gfs_fcst_seg0_validate ...   Passed    0.02 sec

75% tests passed, 1 tests failed out of 4

Label Time Summary:
C48_ATM          =   6.96 sec*proc (4 tests)
gfs_fcst_seg0    =   6.96 sec*proc (4 tests)

Total Test time (real) =   6.98 sec

The following tests FAILED:
	  3 - test_C48_ATM_gfs_fcst_seg0_execute (Failed)
Errors while running CTest
ERROR 8

RUNTESTS/ is not being created at all

aerorahul · 2025-01-16T17:20:20Z

@WalterKolczynski-NOAA
Tests for you will fail until we have a central staged directory.
It requires either;

A central CM provided baseline staged experiment COM, or
User generated baseline from develop
This PR is not sufficient for users to simply do ctest. More work is needed for staging the appropriate input files for automation to kick in.

aerorahul

looks good.

WalterKolczynski-NOAA · 2025-01-16T19:29:17Z

@WalterKolczynski-NOAA Tests for you will fail until we have a central staged directory. It requires either;

A central CM provided baseline staged experiment COM, or

User generated baseline from develop
This PR is not sufficient for users to simply do ctest. More work is needed for staging the appropriate input files for automation to kick in.

Okay, I ran on the same machine so I was expecting it to stage from Terry's space.

Terry McGUinness and others added 30 commits December 6, 2024 10:55

added ctest dir with cmake to create an experment

987ae5b

added CMakeLists for creating initial case EXPDIR usint ctest

61fcf83

ready to test getting the job names

f5f6595

move all operations of creating EXPDIR cases into configure phase

cbe3bab

got cmake configure to work for EXPDIR

6a50e14

can not do both file and variable save for execute_process

bcebeca

working cmake config up to output of job names

be7a890

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

2811629

loop over jobnames and create an experment for each

70a8183

update JOB name update in loop

72e5a9c

addded debug outputs

1f1fcd6

modifying plslot list to be only on a per case

ee27b02

filter pslot list for only relevent cases

bc76ddf

refining get_job_names

38838a6

modigied get_jobs_names to task and metatask names

d18ad01

got better jobname parcer working

ef0e066

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

519a919

replaced generate_workflows with create_experment

f813a1b

removed non-existing ENVIRONMENT setting to consolated COMMAND line

8238228

got cmake script to use creat_experiment

bf579ee

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

28ef1af

just getting started with addeding a ctest

374927f

add in test for configured project

8b4fc4f

added ctest and testing

ce37d86

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

d4e3266

moved ctests dir to top HOMEgfs/ci/ctests with its own scripts dir

bb12cda

added all jobs for tests for each case

f78181c

added data stagging dependances and started it as a stub script

46cddb7

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

d6c3a49

added driver for creating ctests from yaml files

434f869

TerrenceMcGuinness-NOAA marked this pull request as ready for review January 14, 2025 19:14

Terry McGuinness and others added 5 commits January 14, 2025 13:20

included instructions on how to add a new test in the README.md file

b7f80cb

Update README.md

898fcfe

update last bit

Update README.md

12cf0cf

fix misspelled work with create

Update README.md

eeb744f

fixed a grammar in README.md

Update README.md

73211bc

path to source as HOMEgfs

aerorahul reviewed Jan 14, 2025

View reviewed changes

ctests/CMakeLists.txt Outdated Show resolved Hide resolved

aerorahul reviewed Jan 14, 2025

View reviewed changes

modulefiles/module_gwsetup.orion.lua Outdated Show resolved Hide resolved

aerorahul reviewed Jan 14, 2025

View reviewed changes

ctests/scripts/validate.sh Outdated Show resolved Hide resolved

aerorahul reviewed Jan 14, 2025

View reviewed changes

ctests/scripts/validate.sh Outdated Show resolved Hide resolved

TerrenceMcGuinness-NOAA and others added 7 commits January 14, 2025 17:06

Update ctests/scripts/validate.sh

f18f1ee

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

added validate.sh.in template bash wrapper script

b1a9fd2

added cmake to Hercules gwsetup modulefile

f058450

updated path to validate.sh

eede574

needed to remove bash from the COMMAND line for validate.sh

7da3828

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

5f722e3

allow the build to continue whe ctests can not be created without the…

e48ae52

… needed env vars set

TerrenceMcGuinness-NOAA requested a review from aerorahul January 15, 2025 14:28

Terry McGuinness added 3 commits January 15, 2025 14:57

added inpt_files in the path of stagged file to seperate them from ou…

b691a59

…tput_files

added ouput date into case yaml file

5c8a176

removed output_files from C48_ATM

bfe10a2

Merge branch 'NOAA-EMC:develop' into ctest_jjobs_framework

2833514

aerorahul approved these changes Jan 16, 2025

View reviewed changes

WalterKolczynski-NOAA approved these changes Jan 16, 2025

View reviewed changes

WalterKolczynski-NOAA merged commit 01eeb24 into NOAA-EMC:develop Jan 17, 2025
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI JJOB Tests using CMake #3214

CI JJOB Tests using CMake #3214

TerrenceMcGuinness-NOAA commented Jan 8, 2025 •

edited

Loading

WalterKolczynski-NOAA commented Jan 14, 2025

aerorahul Jan 14, 2025

WalterKolczynski-NOAA commented Jan 16, 2025 •

edited

Loading

aerorahul commented Jan 16, 2025

aerorahul left a comment

WalterKolczynski-NOAA commented Jan 16, 2025

CI JJOB Tests using CMake #3214

CI JJOB Tests using CMake #3214

Conversation

TerrenceMcGuinness-NOAA commented Jan 8, 2025 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

WalterKolczynski-NOAA commented Jan 14, 2025

aerorahul Jan 14, 2025

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Jan 16, 2025 • edited Loading

aerorahul commented Jan 16, 2025

aerorahul left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Jan 16, 2025

TerrenceMcGuinness-NOAA commented Jan 8, 2025 •

edited

Loading

WalterKolczynski-NOAA commented Jan 16, 2025 •

edited

Loading