Fix issue 152

[charlienegri]

Summary: fixes #152

Related issue:

Suggested reviewer(s):

Reviewer checklist:

• The headers of all files contain a reference to the repository license
• 100% test coverage of new code - meaning:
  • The overall test coverage increased or remained the same as before
  • Every function is accompanied with a test suite
  • Tests are both positive (testing that the function work as intended with valid data) and negative (testing that the function behaves as expected with invalid data, e.g., that correct exceptions are thrown)
  • Functions with optional arguments have separate tests for all options
• Examples are supported by doctests
• All tests are passing
• All names (e.g., files, classes, functions, variables) are explicit
• Documentation (as docstrings) is complete and understandable

The checklist is based on the S-ENDA conventions and definition of done (see General Conventions). The above points are not necessarily relevant to all contributions. In that case, please add a short explanation to help the reviewer.

[charlienegri]

managed 100% coverage I hope it makes sense

[charlienegri]

added a return (+ test) that should fix #156

[johtoblan]

LGTM

[magnarem]

I checked out this branch and run the pytest and code-coverage tests, and everything sucess and 100%.

However, when I try to ingest some mmd using catalog-rebuilder which is have <mmd:related_dataset relation="parent"> that is just uuid without namespace, I get some exceptions:

INFO:werkzeug:127.0.0.1 - - [20/Apr/2023 15:13:58] "POST /v1/insert HTTP/1.1" 500 -
ERROR    Malformed parent dataset identifier [b'f6cbb81c-1ce1-4080-b242-819e59cee78d']
ERROR:dmci.api.worker:Malformed parent dataset identifier [b'f6cbb81c-1ce1-4080-b242-819e59cee78d']
ERROR    Exception on /v1/insert [POST]
Traceback (most recent call last):
  File "/home/magnarem/anaconda3/envs/senda/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/magnarem/anaconda3/envs/senda/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/magnarem/anaconda3/envs/senda/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/magnarem/anaconda3/envs/senda/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/magnarem/git/s-enda/metno/discovery-metadata-catalog-ingestor/dmci/api/app.py", line 67, in post_insert
    msg, code = self._insert_update_method_post("insert", request)
  File "/home/magnarem/git/s-enda/metno/discovery-metadata-catalog-ingestor/dmci/api/app.py", line 132, in _insert_update_method_post
    valid, msg, data_ = worker.validate(data)
ValueError: not enough values to unpack (expected 3, got 2)
ERROR:dmci.api.app:Exception on /v1/insert [POST]
Traceback (most recent call last):
  File "/home/magnarem/anaconda3/envs/senda/lib/python3.10/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/magnarem/anaconda3/envs/senda/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/magnarem/anaconda3/envs/senda/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/magnarem/anaconda3/envs/senda/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/home/magnarem/git/s-enda/metno/discovery-metadata-catalog-ingestor/dmci/api/app.py", line 67, in post_insert
    msg, code = self._insert_update_method_post("insert", request)
  File "/home/magnarem/git/s-enda/metno/discovery-metadata-catalog-ingestor/dmci/api/app.py", line 132, in _insert_update_method_post
    valid, msg, data_ = worker.validate(data)
ValueError: not enough values to unpack (expected 3, got 2)
127.0.0.1 - - [20/Apr/2023 15:13:58] "POST /v1/insert HTTP/1.1" 500 -
INFO:werkzeug:127.0.0.1 - - [20/Apr/2023 15:13:58] "POST /v1/insert HTTP/1.1" 500 -

Looks like there are missing return parameter on line 125 in worker.py. Should probably return data, as discussed in #156. Also line 85 in worker.py should return 3 variables with data as the last one. (Looks like no tests are actually covering this statement?)

[charlienegri]

I added the check at line 125 for pure redundancy, because we said the fact that a parent does not have a correct identifier should not be possible once we fix py-mmd-tools... but apparently we need it 😅 I will edit the return, you are right that it misses the 3rd argument
the test for line 85 was written to expect a return of 2 parameters as far as I can see:

    # Invalid data format
    passData = readFile(passFile)
    assert passWorker.validate(passData) == (False, "Input must be bytes type")

(and similar for the test of line 125, written to expect 2 params in return) I will fix this too

[charlienegri]

@magnarem can you try again?

[magnarem]

Tested locally with a few test-mmd files. Works ok now.
Updated the dockstring.

[magnarem]

Reviewed and approved

[magnarem]

NOTE!!!!!!

If we merge this issue now, before solving #242 (py-mmd-tools), all incoming MMD files that have <mmd:related_dataset relation="parent"> will be rejected and not stored in the mmd-xml-dev archive

Reason: Input MMD XML file has no valid uri:UUID metadata_identifier

So maybe we sould discuss tomorrow morning meeting if we should solve #242 before we merge this. Or if we are ok with new incoming data not stored in the archives, then we can merge when we want.

[mortenwh]

Sounds best to wait :)

[magnarem]

Approving this MR. But we should probably still wait until #242-py-mmd-tools have been merged.