This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
using_singularity [2020/02/25 15:08] root |
using_singularity [2022/01/26 10:28] (current) root |
||
---|---|---|---|
Line 1: | Line 1: | ||
===== Using Singularity ===== | ===== Using Singularity ===== | ||
- | Singularity is a container mechanism designed for use on a compute cluster. It gives the user very flexible control of their environment (even to which Linux distribution they want to use), but still allows use of the cluster storage, fast network, mutliple nodes, etc. A container is like a virtual machine but is considerably more light-weight: | + | These notes apply to Singularity version 3. The cluster currently has version 3.5.2 installed. |
+ | |||
+ | Singularity is a container mechanism designed for use on a compute cluster. It gives the user very flexible control of their environment (even up to which Linux distribution they want to use), but still allows use of the cluster storage, fast network, mutliple nodes, etc. A container is like a virtual machine but is considerably more light-weight: | ||
Singularity web page: https:// | Singularity web page: https:// | ||
Line 10: | Line 12: | ||
For example, I installed VirtualBox on my Windows 10 laptop, then created a virtual machine running Centos 7. (If VirtualBox is only giving you the choice of 32-bit virtual machines, you need to turn on virtualization support and VTd support in the BIOS of your machine.) | For example, I installed VirtualBox on my Windows 10 laptop, then created a virtual machine running Centos 7. (If VirtualBox is only giving you the choice of 32-bit virtual machines, you need to turn on virtualization support and VTd support in the BIOS of your machine.) | ||
+ | |||
+ | We can also provide access to a Linux virtual machine with root permissions if you need it. Please contact the system administrators to get access to this VM. | ||
==== Installing Singularity ==== | ==== Installing Singularity ==== | ||
- | For Singularity version 3 installation is a little involved because it requires the Go programming language. Probably best to follow the instructions here: https:// | + | For Singularity version 3, installation is a little involved because it requires the Go programming language. Probably best to follow the instructions here: https:// |
==== Creating a Container ==== | ==== Creating a Container ==== | ||
Line 23: | Line 27: | ||
</ | </ | ||
- | Would build you a container with the latest version of Ubuntu (available in the library - 18.10 at the time of writing). Similarly, you could specify " | + | Would build you a container with the latest version of Ubuntu (available in the library - 18.10 at the time of writing). Similarly, you could specify " |
But note that this gets you an **immutable** container: you cannot update it. To get a container that you can update you must create it using the " | But note that this gets you an **immutable** container: you cannot update it. To get a container that you can update you must create it using the " | ||
Line 62: | Line 66: | ||
singularity build container.sif container | singularity build container.sif container | ||
</ | </ | ||
+ | |||
+ | (This step can take quite a long time if a lot has been installed into the sandbox.) | ||
Then copy the .sif file to the cluster. Once there you can run a program inside the container by using the exec command. For instance: | Then copy the .sif file to the cluster. Once there you can run a program inside the container by using the exec command. For instance: | ||
Line 104: | Line 110: | ||
pip install nbresuse | pip install nbresuse | ||
# install system monitoring extensions | # install system monitoring extensions | ||
- | pip install jupyterlab-nvdashboard | + | jupyter labextension install jupyterlab-topbar-extension jupyterlab-system-monitor |
- | jupyter labextension install jupyterlab-topbar-extension jupyterlab-system-monitor | + | |
</ | </ | ||
Line 130: | Line 135: | ||
</ | </ | ||
- | Create | + | Create |
< | < | ||
Line 149: | Line 154: | ||
==== Using Remote Builder ==== | ==== Using Remote Builder ==== | ||
- | If you don’t have root access to a Linux machine you can build a singularity | + | If you don’t have root access to a Linux machine you can build a Singularity container |
- | Go to the Remote Builder website and copy and paste the recipe below into “Build a Recipe” box. Once the build recipe box has been checked click the build button. | + | Go to the Remote Builder website and copy and paste the recipe below into the “Build a Recipe” box. Once the build recipe box has been checked click the build button. |
Additional recipe details can be found here: https:// | Additional recipe details can be found here: https:// | ||
Line 183: | Line 188: | ||
- | ==== Another Example ==== | + | ==== Another Example |
- | We will install | + | We will install |
< | < | ||
- | yum update | + | cd /home/YOU |
- | yum install wget | + | mkdir singularity-images |
- | yum install util-linux | + | cd singularity-images |
- | yum install bzip2 | + | singularity build --sandbox ubuntu library:// |
- | yum install libXext | + | singularity shell --writable --no-home ubuntu |
- | yum install libSM | + | </ |
- | yum install | + | |
+ | You will get a warning saying that / | ||
+ | |||
+ | < | ||
+ | apt update | ||
+ | apt install | ||
mkdir /project | mkdir /project | ||
cd /project | cd /project | ||
Line 201: | Line 211: | ||
./ | ./ | ||
export PATH=/ | export PATH=/ | ||
- | conda create -n qiime1 python=2.7 qiime matplotlib=1.4.3 mock nose -c bioconda | + | source / |
+ | wget https:// | ||
+ | conda env create -n qiime2-2019.10 --file qiime2-2019.10-py36-linux-conda.yml | ||
</ | </ | ||
- | Or within an Ubuntu " | + | Copy the code added to the end of / |
< | < | ||
- | apt-get update | + | chmod ugo+x / |
- | apt-get upgrade | + | |
- | apt-get install wget | + | |
- | apt-get install libglib2.0-0 | + | |
- | apt-get install bzip2 | + | |
- | apt-get install libxext6 | + | |
- | apt-get install libsm6 | + | |
- | apt-get install libxrender1 | + | |
</ | </ | ||
- | Create a script, | + | Create a script, |
+ | |||
+ | < | ||
+ | apt install nano | ||
+ | </ | ||
+ | |||
+ | The two mysterious **exports** at the start of the run-qiime file come from error messages generated if you don't include these (the errors suggest them as a solution). | ||
< | < | ||
#!/bin/bash | #!/bin/bash | ||
+ | export LC_ALL=C.UTF-8 | ||
+ | export LANG=C.UTF-8 | ||
+ | / | ||
export PATH=/ | export PATH=/ | ||
- | source activate | + | source activate |
- | print_qiime_config.py -t | + | qiime --help |
</ | </ | ||
< | < | ||
- | chmod go+x run-qiime | + | chmod ugo+x run-qiime |
</ | </ | ||
Line 233: | Line 247: | ||
< | < | ||
- | singularity exec container.img | + | singularity exec --writable ubuntu |
</ | </ | ||
- | Copy the container to the cluster, and run the exec command again. | + | (You need --writable or you will get some errors about "read-only filesystem".) |
+ | |||
+ | Generate a .sif file from the ubuntu | ||
+ | |||
+ | < | ||
+ | singularity build qiime2.sif ubuntu | ||
+ | </ | ||
+ | Copy the .sif file to the cluster, and run the exec command on it. You can write a script that you can " | ||