BRC Cluster Workshop

User Tools

Site Tools

Trace: • using_singularity
using_singularity

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
using_singularity [2020/02/25 15:45]
root 		using_singularity [2022/01/26 10:28] (current)
root
Line 1: Line 1:
 ===== Using Singularity ===== ===== Using Singularity =====
  
-Singularity is a container mechanism designed for use on a compute cluster. It gives the user very flexible control of their environment (even to which Linux distribution they want to use), but still allows use of the cluster storage, fast network, mutliple nodes, etc. A container is like a virtual machine but is considerably more light-weight: it uses the kernel that is running on the machine on which it is executed rather than including its own. Singularity containers can be run essentially like ordinary programs, so they can be submitted as jobs on the cluster (to one or more nodes).+These notes apply to Singularity version 3. The cluster currently has version 3.5.2 installed. 
 + 
 +Singularity is a container mechanism designed for use on a compute cluster. It gives the user very flexible control of their environment (even up to which Linux distribution they want to use), but still allows use of the cluster storage, fast network, mutliple nodes, etc. A container is like a virtual machine but is considerably more light-weight: it uses the kernel that is running on the machine on which it is executed rather than including its own. Singularity containers can be run essentially like ordinary programs, so they can be submitted as jobs on the cluster (to one or more nodes).
  
 Singularity web page: https://sylabs.io/docs/ Singularity web page: https://sylabs.io/docs/
Line 10: Line 12:
  
 For example, I installed VirtualBox on my Windows 10 laptop, then created a virtual machine running Centos 7. (If VirtualBox is only giving you the choice of 32-bit virtual machines, you need to turn on virtualization support and VTd support in the BIOS of your machine.) For example, I installed VirtualBox on my Windows 10 laptop, then created a virtual machine running Centos 7. (If VirtualBox is only giving you the choice of 32-bit virtual machines, you need to turn on virtualization support and VTd support in the BIOS of your machine.)
 +
 +We can also provide access to a Linux virtual machine with root permissions if you need it. Please contact the system administrators to get access to this VM.
  
 ==== Installing Singularity ==== ==== Installing Singularity ====
  
-For Singularity version 3 installation is a little involved because it requires the Go programming language. Probably best to follow the instructions here: https://sylabs.io/guides/3.5/admin-guide/installation.html# or here: https://sylabs.io/guides/3.5/user-guide/quick_start.html+For Singularity version 3, installation is a little involved because it requires the Go programming language. Probably best to follow the instructions here: https://sylabs.io/guides/3.5/admin-guide/installation.html# or here: https://sylabs.io/guides/3.5/user-guide/quick_start.html
  
 ==== Creating a Container ==== ==== Creating a Container ====
Line 23: Line 27:
 </code> </code>
  
-Would build you a container with the latest version of Ubuntu (available in the library - 18.10 at the time of writing). Similarly, you could specify "centos" here (and get Centos 7.6). +Would build you a container with the latest version of Ubuntu (available in the library - 18.10 at the time of writing). Similarly, you could specify "centos" here (and get Centos 8.4). 
  
 But note that this gets you an **immutable** container: you cannot update it. To get a container that you can update you must create it using the "--sandbox" flag. This creates the container as a directory on your system rather than as a single file. There is also another technique you can use to (virtually) make changes to your container: an overlay. This is an overlay filesystem which you add on top of the container. The contents of that overlay get changed not the container itself. This technique is a little more complicated, but can be used to allow a non-root user to make changes (to the overlay). But note that this gets you an **immutable** container: you cannot update it. To get a container that you can update you must create it using the "--sandbox" flag. This creates the container as a directory on your system rather than as a single file. There is also another technique you can use to (virtually) make changes to your container: an overlay. This is an overlay filesystem which you add on top of the container. The contents of that overlay get changed not the container itself. This technique is a little more complicated, but can be used to allow a non-root user to make changes (to the overlay).
Line 62: Line 66:
 singularity build container.sif container singularity build container.sif container
 </code> </code>
 +
 +(This step can take quite a long time if a lot has been installed into the sandbox.)
  
 Then copy the .sif file to the cluster. Once there you can run a program inside the container by using the exec command. For instance: Then copy the .sif file to the cluster. Once there you can run a program inside the container by using the exec command. For instance:
Line 104: Line 110:
 pip install nbresuse pip install nbresuse
 # install system monitoring extensions # install system monitoring extensions
-pip install jupyterlab-nvdashboard +jupyter labextension install jupyterlab-topbar-extension jupyterlab-system-monitor
-jupyter labextension install jupyterlab-topbar-extension jupyterlab-system-monitor jupyterlab-nvdashboard+
 </code> </code>
  
Line 211: Line 216:
 </code> </code>
  
-Copy the code added to the end of /root/.bashrc to initialize conda to a file named /project/start-conda. Make this file executable. +Copy the code added to the end of /root/.bashrc to initialize conda to a file named /project/start-conda. Make this file executable by all users.  
 + 
 +<code> 
 +chmod ugo+x /project/start-conda 
 +</code>
  
 Create a script,run-qiime, to execute qiime (in /project). Set the script to be executable by group and others. To do this you will need an editor! Create a script,run-qiime, to execute qiime (in /project). Set the script to be executable by group and others. To do this you will need an editor!
Line 232: Line 241:
  
 <code> <code>
-chmod go+x run-qiime+chmod ugo+x run-qiime
 </code> </code>
  
Line 238: Line 247:
  
 <code> <code>
-singularity exec container.img /project/run-qiime+singularity exec --writable ubuntu /project/run-qiime
 </code> </code>
  
-Copy the container to the cluster, and run the exec command again. You can write a script that you can "sbatch" to run your container on the compute nodes.+(You need --writable or you will get some errors about "read-only filesystem".) 
 + 
 +Generate a .sif file from the ubuntu container: 
 + 
 +<code> 
 +singularity build qiime2.sif ubuntu 
 +</code>
  
 +Copy the .sif file to the cluster, and run the exec command on it. You can write a script that you can "sbatch" to run your container on the compute nodes.
  
using_singularity.1582663522.txt.gz · Last modified: 2020/02/25 15:45 by root

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: Public Domain
Public Domain Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki