Proposed enhancement for command-line interface tests

[mwtoews]

This is an issue that I intend to resolve, but need to consult for feedback before starting.

The issue is that bash scripts like test/cli/testproj are only run/tested for UNIX (i.e. here), which means these are not tested for Windows. Probably the most portable way to run these for all platforms is to make a Python script to do this, which shouldn't be an issue since Python is already used for building the data. More importantly, the tests would need a structured data format, which may contain arguments, input line(s) and expected output line(s).

One obvious candidate format is JSON:

[
  {
    "args": "proj +ellps=WGS84 +proj=ob_tran +o_proj=latlon +o_lon_p=0.0 +o_lat_p=90.0 +lon_0=360.0 +to_meter=0.0174532925199433 +no_defs -E -f %.3f",
    "in": "2 49",
    "out": "2 49\t2.000\t49.000"
  },
  {
    "comment": "Test CRS option",
    "args": "proj EPSG:32620 -S",
    "in": "-63 0",
    "out": "500000.00\t0.00\t<0.9996 0.9996 0.9992 0 0.9996 0.9996>"
  },
  {
    "comment": "Test projection factors on projected CRS with non-Greenwhich prime meridian",
    "args": "proj EPSG:27571 -S",
    "in": "2.33722917 49.5",
    "out": "600000.00\t1200000.00\t<0.999877 0.999877 0.999755 0 0.999877 0.999877>"
  },
  {
    "comment": "Test projection factors on compound CRS with a projected CRS",
    "args": "proj EPSG:5972 -S",
    "in": "9 0",
    "out": "500000.00\t0.00\t<0.9996 0.9996 0.9992 0 0.9996 0.9996>"
  }
]

or possibly nicer as more compact and versatile YAML file, which can easily split tags like "exe" at the top:

exe: proj
tests:
- args: +ellps=WGS84 +proj=ob_tran +o_proj=latlon +o_lon_p=0.0 +o_lat_p=90.0
    +lon_0=360.0 +to_meter=0.0174532925199433 +no_defs -E -f %.3f
  in: 2 49
  out: "2 49\t2.000\t49.000"
- comment: Test CRS option
  args: EPSG:32620 -S
  in: -63 0
  out: "500000.00\t0.00\t<0.9996 0.9996 0.9992 0 0.9996 0.9996>"
- comment: Test projection factors on projected CRS with non-Greenwhich prime meridian
  args: EPSG:27571 -S
  in: 2.33722917 49.5
  out: "600000.00\t1200000.00\t<0.999877 0.999877 0.999755 0 0.999877 0.999877>"
- comment: Test projection factors on compound CRS with a projected CRS
  args: EPSG:5972 -S
  in: 9 0
  out: "500000.00\t0.00\t<0.9996 0.9996 0.9992 0 0.9996 0.9996>"

versatility is more important with longer multi-line input/output from test/cli/testvarious and test/cli/tv_out.dist

As for the specification of the input tests, there would be required and optional keys. I'd need to sort these specifications after looking at all of the test files.

After a Python script is built to run each to-be-converted file, it would then be called from CTest (replacing proj_add_test_script_sh). I may need to consider skipping these tests if a suitable YAML parser is not available for Python, since unfortunately there is no standard module for YAML (unlike JSON).

Should Python and YAML test data be used for a modernized CLI testing framework? Is there anything else to consider?

[hobu]

Should Python and YAML test data be used for a modernized CLI testing framework? Is there anything else to consider?

I would vote JSON even though I would rather edit YAML for this reason – these tests are also likely to be conformance definitions for other libraries like Proj4js or Proj4J, and since PROJ is the primary definition of how and whether something should work, this is the most convenient way for our project to communicate that to others. JSON is is certainly more prevalent in JavaScript and the availability of JSON tools in just about any serious development language is not in doubt whereas it sometimes is with YAML.

[kbevers]

Should Python and YAML test data be used for a modernized CLI testing framework? Is there anything else to consider?

I am absolutely in favour of doing this. Great suggestion! My only hesitation would be to use yaml. As you point out it's not part of the Python standard library. My gut feeling is that it would be smart to keep dependencies outside the standard library to a minimum.

[mwtoews]

Doing a bit of git archaeology, I see that these tests originally in nad/ were some of the original files from Gerald and appear to the only test methods used at the time.

As for JSON to potentially be used by related libraries to test their capabilities, this is a good point. However tools like yq can be used to convert YAML to JSON, if needed. Also, I'm still unsure what it a multi-line in/out JSON would look like, and I'd imagine it would be more difficult to format new CLI tests. YAML is much easier to create new tests.