Simplify pipeline configuration

[donkirkby]

Background

I've recently been adding support for optional arguments as part of #511 and thinking about switching from Docker to Singularity for #737. One of the main enhancements to the current milestone is #723 to make it easier to create a new method by not defining the outputs.

All of this has got me thinking about how complex the code is for configuring a new pipeline and executing a pipeline in a run. It took over a week to add support for optional arguments, and I still have to make changes to the user interface and API for launching runs. The complexity also makes Kive slow at run time, as it searches for results that it can reuse.

We have learned a lot about new tools since we started the Kive project: Slurm, GitHub/GitLab, and Docker/Singularity. Could we replace a lot of the complexity in Kive with features from these tools? We did this in the past when we moved from MPI to Slurm.

This issue is a proposal for the team to discuss - a plan to simplify Kive.

Features to Remove

These are the main sources of complexity. Are they worth it?

• Reusing old results.
  • Because of purging, it doesn't get used that much.
  • Takes a lot of code!
  • The place it does get used is restarting a run batch. Maybe we could support that scenario with a search function that finds a run by its inputs.
• Nested pipelines
  • More flexible to handle this outside of Kive with calls to the API.
• Custom cables
  • Can handle this as another step in a pipeline script.
• Compound data types and validation
  • Adds an extra step after the method runs.
  • Makes the pipeline configuration more complicated.

Features to Extract

• Linking multiple methods into a pipeline
  • Upload a zip/tar.gz file with scripts in it.
  • Use the pipeline UI to wire inputs and outputs between scripts.
  • Save pipeline as a pipeline.json file in the tar.gz file with the scripts.
  • Launch the pipeline with a default Singularity image, plus the tar.gz file mounted on /mnt/bin, plus a driver script that reads the pipeline.json file and executes the steps.
  • The advantage is that this all runs as a single Slurm job, and the accounting is much simpler.
  • You can also migrate a pipeline from development to test to production by downloading the tar.gz file that includes the pipeline.json, and uploading it to another Kive server. A Kive container can be either a Singularity image or a tar.gz file. The tar.gz file is a layer on top of the default Singularity image.
  • When you upload a new tar.gz container without a pipeline.json, it can copy the pipeline.json from the previous revision in the container family.

Features to Keep

These features are working well, and I think they're the main value of Kive.

• Distribute work across compute nodes with Slurm.
• Isolate sets of dependencies with Singularity. (Docker handles the dependencies well, but doesn't work on the cluster. Experiments with Singularity look very promising.)
• Find results by MD5. This was the original vision for Kive: to trace a result file back to the version of the software and the data that produced it.
• Reproduce an old result, typically to look at some related intermediate results.

Features to Add

• Link to Git comparison of different pipeline versions.

• Define pipeline configuration in Singularity labels. That way, you could migrate a pipeline from development to test to production by copying the image file and nothing else.

    KIVE_INPUTS=name1 name2 --option_name --multiple_option_name*
    KIVE_OUTPUTS=name1 name2 directory_name/ --output_option
  

Task List

• Add ContainerArgument table, and configure on container page.
• Add ContainerRun and related tables.
• Copy analysis page to create a container run.
• Display container runs.
• Create inputs for container run.
• Add priority to container run.
• Launch a container run and record outputs.
• View or download outputs.
• Handle failed run, and record stderr/stdout.
• Launch singularity with app name.
• Set memory and CPU limits in Slurm.
• View or edit batch.
• Cancel a run.
• Include outputs and other runs in a run's removal plan.
• Don't remove active runs.
• Find container runs by container and inputs.
• Add missing filters to search pages.
• Handle stdout/stderr longer than 2KB.
• Document how to create a Singularity image.
• Purge sandboxes. Janitor tasks for container runs #750
• Purge log files. Janitor tasks for container runs #750
• Purge output datasets. Janitor tasks for container runs #750
• Container as an archive of scripts, with pipeline configuration in a JSON file. Run a pipeline in a container #751
• Remove old features. Remove old runs and pipelines #752

[donkirkby]

I sketched out how the new container app might look:

This would eventually replace the apps for archive, method, pipeline, transformation, some of librarian, and possibly metadata and datachecking.

[donkirkby]

After team discussion, the plan is to switch to singularity first by completing #737, then tackle this simplification.
We could keep the pipeline assembly tools, but use them to generate a zip file or Singularity image that then runs like any other pipeline.