Local additional code

[dafyddstephenson]

-> This PR branches from #52 , do not merge until merging #52 <-

Closes #31.

Summary

This PR introduces support for local additional code, as well as improving the handling of model namelists that are to be modified by C-Star (the latter changes being necessary to test the former). Local AdditionalCode instances should be created using a path to a directory, instead of a remote repository, but the structure of this directory should be the same:

    <additional_code_dir>                                                                                                                                                                                                                                                                          
       ├── namelists                                                                                                                                                                                                                                                                               
       |      └ <base_model_name>                                                                                                                                                                                                                                                                  
       |              ├ <namelist_file_1>                                                                                                                                                                                                                                                          
       |              |       ...                                                                                                                                                                                                                                                                  
       |              └ <namelist_file_N>                                                                                                                                                                                                                                                          
       └── source_mods                                                                                                                                                                                                                                                                             
              └ <base_model_name>                                                                                                                                                                                                                                                                  
                      ├ <source_code_file_1>                                                                                                                                                                                                                                                       
                      |       ...                                                                                                                                                                                                                                                                  
                      └ <source_code_file_N>      

Meanwhile, calls to AdditionalCode.get() now check whether namelist filenames have _TEMPLATE as a suffix. If so, a new copy without this suffix is created alongside the template file. The template is tracked with AdditionalCode.namelists as before, the copy is tracked using a new attribute AdditionalCode.modified_namelists. This is an improvement on previous handling but not a viable long term solution. I have raised Issue #56 in response.

List of changes

Changes directly required for local additional code support:

• Move DataSource out of base/input_dataset.py and into dedicated module base/datasource.py
• Expand DataSource to check if location is a directory or local repository
• Rename AdditionalCode.source_repo:
  • source_repo attribute is renamed to AdditionalCode.source, type changed from str -> DataSource
  • source_repo argument to AdditionalCode.__init__() is renamed to location (str).
    This location is used by __init__ to create a DataSource instance and set the AdditionalCode.source attribute.
• Similarly rename source and source_repo to location in:
  • Case.from_blueprint()
  • Case.persist()
  • cstar_blueprint_roms_marbl_example.yaml
• Rename "local_path" to "local_dir" in AdditionalCode.get to avoid confusion with AdditionalCode.local_path
• Make AdditionalCode.checkout_target optional (in case source is not a repository)
• Update AdditionalCode.get(). This previously cloned a repository to a temporary location and then moved the files needed by the AdditionalCode instance. It now does this only if AdditionalCode.source.location_type is url and AdditionalCode.source.source_type is repository. If location_type is path then the files are simply copied from the provided path.
• The test_roms_marbl_example.py script now also uses local additional code as well as local input datasets in its second (local) stage

Related changes associated with namelist handling:

These changes were necessary to implement the test described above, as previously the template namelist file was edited in place and so couldn't be re-used by a second case.

• Add an attribute AdditionalCode.modified_namelists to keep track of edited namelists based on templates
• Add logic in AdditionalCode.get() that makes copies of any namelist files with the suffix _TEMPLATE to a file without this suffix and adds it to AdditionalCode.modified_namelists, e.g.:
  AdditionalCode.namelists = ["roms.in_TEMPLATE",] -> AdditionalCode.modified_namelists = ["roms.in",]
• Update ROMSComponent.pre_run() and ROMSComponent.run() to reflect this namelist handling

Other/bugfixes:

• Update InputDataset.local_path to point to the symlink created by InputDataset.get() after call (if dataset is local, addressing an inconsistency in Better input dataset hash handling #52 )
• update pre-commit config file to rename cstar_ocean -> cstar
• swap out os for pathlib in additional_code.py and datasource.py
• minor changes to case.py with additional type checking
• Update docstrings, __str__s and example notebook to reflect above changes

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[TomNicholas]

I'm looking through this and so far it looks good. We should probably chat about #56 at some point though.