Update Bulbasaur physical drive settings in the template; finalize Bulbasaur deployment

Updated the settings in the Bulbasaur Ansible configuration variables to reflect the physical drives we have installed in that machine. This tag is what we aim to use to finalize deployment of Bulbasaur.

Change network drive mount permissions; mid-Bulbasaur deployment

This commit is the one used to change the network drive permissions on Octomore as of January 16, 2024.

Other changes have also been included toward doing the deployment on Bulbasaur; this commit contains most of the playbooks used for deploying to Bulbasaur (which is currently mid-deployment as we wait for a backup drive to be delivered).

Migration of Octomore to Ubuntu 22.04LTS

This commit contains the documentation, scripts, and Ansible playbooks used in migrating Octomore to Ubuntu 22.04LTS.

Python 3

The main change is #721 updating to Python 3. Other changes include:

• #785, #786, and #782 improve error handling when launching Slurm jobs.
• #774 and #793 fix permissions bugs.
• #833 removes PostgreSQL 9 left by the previous release.

See the milestone for a complete list.

Config Changes

• Install Python 3.6 and switch Apache to launch it.

  sudo systemctl stop httpd
  sudo yum install centos-release-scl
  sudo yum install python36 python36-devel rh-python36-mod_wsgi
  sudo cp /opt/rh/httpd24/root/usr/lib64/httpd/modules/mod_rh-python36-wsgi.so /lib64/httpd/modules
  sudo cp /opt/rh/httpd24/root/etc/httpd/conf.modules.d/10-rh-python36-wsgi.conf /etc/httpd/conf.modules.d
  sudo mv /opt/venv_kive /opt/py27_venv_kive
  sudo python3.6 -m venv /opt/venv_kive
  sudo /opt/venv_kive/bin/python -m pip install --upgrade pip
  sudo /opt/venv_kive/bin/python -m pip install -r requirements.txt
  sudo sed -ie 's/python2.7/python3.6/' /etc/httpd/conf.d/kive.conf
  sudo systemctl start httpd
  

• Add KIVE_SLURM_PATH entry to /etc/sysconfig/httpd.

• Remove Python 2's .pyc files with find . -name '*.pyc' -delete.

• Remove PostgreSQL 9 with sudo yum remove postgresql-libs. That will take the other packages that depend on it.

• The Python packages in Kive's virtual environment should already be up to date, because you just installed them. That should include the switch to psycopg2-binary.

• Rewire all the PostgreSQL 10 commands: alternatives --list|grep pgsql|sed -e "s/\t.*//"|sudo xargs -n 1 alternatives --auto

• Check that worked by running these two commands. If you get a help message, they're installed.

  psql --help
  pg_dumpall --help
  

• Export the /usr/sbin folder from the head node to the compute nodes, by including it in /etc/beowulf/fstab. After rebooting the compute nodes, it's working if this command doesn't hang anymore:

   bpsh 0 python3.6 -c "import uuid"
  

Remove old pipelines

This release is mostly for #752 to remove old runs and pipelines. Other issues:

• #747 switch to pytest
• #784 retry after sbatch failure

See the milestone for the full list of changes.

Extra deployment steps:

• Run the migrations and the find_orphans management command as part of #752.
• Delete the CodeResources, Logs, and Sandboxes folders.
• Upgrade requirements.

Run a pipeline in a container

The main change is #751 to run a pipeline in a container, but there are other changes:

• #754 Make reruns easier
• #762 and #766 error handling on purge task

See the milestone for a full list of issues.

Configuration changes:

• Upgrade Django and Django REST framework from requirements.txt.
• Upgrade PostgreSQL from 9 to 10, for performance. Set user access in data/pg_hba.conf and listening addresses in data/postgresql.conf.
• #751 migrations
• #710 slurm accounting purge
• #765 Apache log rotation
• #749 flash messages support in Django settings file
• #751 run convert_pipelines command

Simplify Pipeline Configuration

The main issue is #739 to simplify pipeline configuration by putting it all inside a singularity container.

Other major changes:

• #750 janitor tasks
• #735 switch configuration details to environment variables

For a full list of issues, see the milestone.

This release requires a lot of configuration changes, but they can follow the vagrant script. One extra step:

• Load the slurm module into the apache process by adding this to /etc/sysconfig/httpd:

  # Load the slurm module.
  LD_LIBRARY_PATH=/opt/scyld/slurm/lib64
  PATH=/opt/scyld/slurm/sbin:/opt/scyld/slurm/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
  

Replace Docker with Singularity

The main change is #737 to replace Docker with Singularity, because Docker doesn't work well on our computing cluster. The other important change is to upgrade Django to 1.11 in #720.

This release requires some extra steps:

• install singularity, including changes to /etc/beowulf/fstab
• upgrade Django to 1.11 in system Python and fleet virtualenv
• compare settings.py, and make sure to add container app
• run the migrations for Singularity images
• upload Singularity images
• run the convert_docker_images management command

For the full list of changes, see the milestone.

Support Slurm 17

The main change is issue #726 to support updating Slurm to version 17. This comes along with ClusterWare upgrades related to issue #709.

Docker support

The main change is #681: docker support in pipelines.

See the milestone for a full list.