Ctrl+K

Build a Cloud VM

Build a Cloud VM#

In the following, we explain how to install Quantum Mobile on your own virtual machine in the cloud.

This procedure is automated entirely using an ansible playbook in the quantum mobile repository. It has been used successfully to deploy Quantum Mobile on servers from Amazon Web Services, Google Compute Cloud, Huawei Cloud, Openstack as well as on bare metal servers.

Note

This process essentially a sub-set of the process to build the Desktop Edition (see the instructions here), in that we are provisioning an existing VM, rather than creating one from scratch.

Prerequisites#

Server#

  • A server running Ubuntu 18.04 LTS Can be hardware or virtual machine (tested on OpenStack, Amazon Web Services and Huawei Cloud).

  • At least 12GB disk size (including Ubuntu); better 15GB or more. Note: After cleaning temporary files, QM occupies ~6GB of disk space.

  • Access to server via SSH key as user with sudo rights

Security rules:

  • Port 22 open (for SSH access)

  • You may want to open further ports (optional):

    • port 8888 to connect to the AiiDA Lab Server (or value of aiidalab_jupyter_port)

    • port 8890 to connect to thr Jupyter Lab Server (or value of aiida_jupyter_port)

    • port 5000 to connect to the AiiDA REST API

Client (your computer)#

Provisioning the server#

To get set up, run the following on your client (e.g. your laptop – not on the server itself):

git clone https://github.com/marvel-nccr/quantum-mobile.git
cd quantum-mobile

The inventory.yml already contains templates for connecting to servers from different cloud providers, which are easy to populate with the connection details for your server. For example, to connect to an aws host, adapt:

    aws:
      cloud_platform: aws
      ansible_host: 34.250.68.129 # change this
      ansible_ssh_common_args: -i ./keys/quantum-mobile.pem -o StrictHostKeyChecking=no
      ansible_user: ubuntu

with:

  • the IP of the server

  • the path to your private SSH key for connecting to the server

  • the user to connect as via SSH

See also

The ansible inventory and playbook variables documentation.

You may also want to “tune” the variables under vars: in inventory.yml, such as:

  • vm_user: the user for which to install the simulation environment (usually not the admin user you are connecting as)

  • vm_memory, vm_cpus

  • If you want to pre-load an SSH public key for the vm_user to connect with (as opposed to the admin key set above), uncomment and set the add_user_public_key variable.

Finally, to run the ansible playbook against the required host, use:

pip install tox
tox -e ansible -- --extra-vars "build_hosts=aws"

This will take of the order of one hour, after which your server should be fully deployed and operational.

Tip

The ansible steps are idempotent, meaning they will essentially do nothing when run again on the same machine.

This means, if the build was interrupted for any reason, you can simply re-run the above command.

You can log in to the server as the vm_user via the public SSH key you provided.

Saving the image#

Before creating an image from the disk volume of the server you provisioned:

  1. Remove unnecessary temporary build files:

    tox -e ansible -- --extra-vars "build_hosts=aws" --tags cleanup

  2. Clear bash history: SSH to the VM and run cat /dev/null > ~/.bash_history && history -c && exit

  3. Shut down instance (this will clear temporary data in /tmp)

Now follow the instructions of your platform for creating an image from your server.

Platform specific guidance#

AWS#

  • Create new keypair from the GUI - note: this will always be added to the ubuntu user.

  • If you intend to publish the image, follow AWS instructions to remove system SSH keys:
    sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub

  • Listing an image publicly: After creation of the image, edit its properties and select “public”

GCP#

  • GCP lets you create the SSH key-pair by yourself:

    • Create new SSH keypair on your machine.

    • Edit the user@server bit at the end of the public key to user “ubuntu” (GCP detects this user and adds the SSH public key for this user in the VM)

    • add public key to VM

  • VM IP seems to be stable even between stop/start of VM

  • Creating the image

    • GCP distinguishes between “Machine images” (private) and “Images” (can be public). You want the second option.

    • List image publicly: (not sure whether there is a way to do this via the GUI)

      • Click on “cloud shell” logo on the top right

      • Paste command:

        gcloud compute images add-iam-policy-binding quantum-mobile-20-05-0 \
   --member='allAuthenticatedUsers' \
   --role='roles/compute.imageUser'

Troubleshooting#

Slow network connection#

When provisioning machines in remote networks (e.g. Asia), the time needed for establishing an SSH connection can be prohibitively slow. In such cases, the “mitogen” connection plugin can lead to dramatic speedups (it uses python remote procedure calls).

In order to enable mitogen, execute the following command:

wget https://networkgenomics.com/try/mitogen-0.2.8.tar.gz && tar xf mitogen-0.2.8.tar.gz

and uncomment the corresponding lines in the ansible.cfg file.

In tests, an example task (adding the “max” user to two new user groups) took 15s on an AWS VM in the hongkong region, and 156s on a Huawei Cloud VM in the Guangzhou region (and that’s after turning on “pipelining” to reduce the number of SSH connections). Switching to mitogen brought down the timings to 1s for the AWS hongkong machine and 3s on the Huawei VM (i.e. more than one order of magnitude).

Contents