Improve CLI testing

[mwtoews]

This enhancement improves command-line interface tests across several platforms.

Until now CLI testing was done using a range of Unix shell scripts, comparing expected output and showing differences on failure. While this approach has worked fine for >20 years, it is limited to Unix shells (i.e. not Windows) and each test's inputs and expected outputs need to be cross-referenced between two files.

The more modern approach in this PR is to use a Python script to run tests described in YAML files. Each file describes a series of tests to run with an executable. Each test may have arguments (args), input (in), expected stdout, stderr, or combined stdout+stderr (out), as well as an expected exitcode (usually 0). There are a few optional components, such as use of a temporary directory (tmpdir) with files to copy; custom file content; the ability to skip tests based on a simple expression (skipif); and output post-processing commands (grep, grep -v, sub, head, tail, sort) used on command output streams. A full description of the format can be made later (after a review).

The Python script is currently located at test/cli/run_cli_test.py along with the YAML test files that have been mostly manually converted from the shell scripts. This script can be run directly, or via ctest. CMake does some checks before to ensure the requirements have been met, otherwise a warning is shown that the tests won't run. The requirements are: Python 3.7+, and one of either PyYAML or ruamel.yaml. Optionally pytest can be used to self-test the Python script.

One important maintainer feature of the script is --update to update the expected result (stdout, stderr, out and/or exitcode). This mode is enabled only with ruamel.yaml, which has round-trip read/write capabilities to preserve YAML styles and comments when re-writing the file. This mode might be needed when (e.g.) updating proj.db.

Some of the older shell script tests are kept, as they have tests that are better kept as shell scripts. The test names in ctest are the same as the input files, and they have been renamed to easier identify.

Closes #4110

[mwtoews]

Note that all of the old tests should still run in some form, but no new tests have been added. New tests can be added later, e.g. for geod/invgeod. Also binary I/O tests can be added later (with skipif: byteorder != "little"), as I've enabled this support. When text I/O is defined in the YAML file it is otherwise utf-8 encoded on all platforms.

Inline comments added.

[rouault]

LGTM. Excellent work!

[mwtoews]

@rouault comment above to anything that I've raised if you have any strong opinions. Mostly I'm unsure how locale set-up and testing should work in test/cli/test_cs2cs_locale.sh.

Also, before merging, I could remove most of the repetition by removing -E echo flags. The expected stdout/out is easy to update with --update.

[rouault]

Mostly I'm unsure how locale set-up and testing should work in test/cli/test_cs2cs_locale.sh.

Your way is probably OK. Otherwise that logic (setting LC_ALL and PROJ_USE_ENV_LOCALE=1 in os.environ) could perhaps be ported to the Python launcher itself to run all tests with a non C-locale if available

Also, before merging, I could remove most of the repetition by removing -E echo flags. The expected stdout/out is easy to update with --update.

Yes, the -E flag doesn't bring much with the new framework.

[mwtoews]

Most tests have removed echo -E flag, except a few specific tests.

Some basic docs have been added (as part of the Python script, for now).

The run_cli_test.py with this PR is generic, and could potentially be adapted for other projects. It could be moved into a stand-alone Pure Python package, if there is interest.

Lastly, a note to packagers (e.g. /~https://github.com/conda-forge/proj.4-feedstock/). While there are no new build deps, there are new recommended test deps. A minimum of Python 3.7 and pyyaml should be good, otherwise most of the CLI tests won't run. A warning will be shown in the first cmake command.

[jjimenezshaw]

Otherwise that logic (setting LC_ALL and PROJ_USE_ENV_LOCALE=1 in os.environ) could perhaps be ported to the Python launcher itself to run all tests with a non C-locale if available

In PDAL, to test that it was working properly with a non classic locale, I did this in the github actions (only in Linux): /~https://github.com/PDAL/PDAL/blob/84d0a9d076d3ccfd9f48dbab609ec093e1121981/.github/workflows/linux.yml#L75
That locale is later enabled in the test suite, and all the tests run again.

[mwtoews]

On locales, I was considering adding this effort to the Python scripting, but it mostly applies to POSIX systems, thus is easier to keep testing the .sh script (i.e., I copied excerpts from testvarious into test_cs2cs_locale.sh). (I still don't fully understand the interactions of various combinations of PROJ_USE_ENV_LOCALE with LC_ALL, as I never see the comma decimal separator).

[rouault]

I still don't fully understand the interactions of various combinations of PROJ_USE_ENV_LOCALE with LC_ALL, as I never see the comma decimal separator

PROJ_USE_ENV_LOCALE=1 commands the call to

PROJ/src/apps/cs2cs.cpp

Line 799 in 1d0354f

setlocale(LC_ALL, "");

, which enables the taking into account of the LOCALE environment variable by various libc functions

as I never see the comma decimal separator

The test checks that we are able to parse inputs in a non-locale aware way, that is we don't use regular atof() for that (and more generally even internally, e.g when ingesting strings from the database to double), that would fail to parse numbers like "1.25" with a locale with a decimal separator that wouldn't be dot.

The shell script is probably good enough for now.