Transferring Millstone Data Between EC2 Instances

The following describes moving Millstone data from machine A to machine B. We use an intermediate EBS volume to transfer the data, leaving the new volume attached to the second instance. Depending on the use case, you may decide to transfer the data to the root EBS of machine B.

The following are the rough steps, although I may have left out details (permissions issues, typos in commands, etc.), so don't assume they work perfectly.

• Create a new EBS volume that is big enough for the data

• Mount EBS on machine A

  • Since you're mounting for the first time, you'll have to format the disk volume to give it a filesystem:

      sudo mkfs -t ext4 /dev/xvdf
    

  • Then mount:

      sudo mkdir /millstone_data
      sudo mount /dev/xvdf /millstone_data
    

• Dump Posgresql database using pg_dump and save this on to the new EBS

    sudo -u postgres pg_dump genome_designer > /millstone_data/genome_designer.backup
  

• Copy the Millstone filesystem (temp_data/) to the EBS. Note that EBS read/write is super slow.

• Unmount EBS from machine A

• Install the Millstone flavor of cloudbiolinux on machine B

• Restart machine B so that it gets bootstrapped

• Mount the transfer volume EBS on machine B

  • DO NOT FORMAT THE DISK THIS TIME

• Swap out the Postgresql genome_designer database on machine B with the pg_dump file from A

    sudo -u postgres psql
    postgres=# DROP DATABASE genome_designer;
    postgres=# CREATE DATABASE genome_designer WITH OWNER genome_designer;
    <exit postgres>
    sudo -u postgres psql genome_designer < /millstone_data/genome_designer.backup
  

• In conf/local_settings.py, set MEDIA_ROOT to point to the new temp_data/ location

• In the jbrowse/ directory, change the gd_data symlink to point to the new temp_data/ location

• Restart Millstone

    sudo supervisorctl restart genome_designer
  

You can now delete the old instance if you'd like.