Utilizing singularity containers in SD-Desktop

As the SD Desktop is not directly connected internet, you can't use tools like Git, Conda or Pip to install new software there. In stead, you can use Singularity Singularity software container tool to add new software to your SD Desktop environment. However, you have to first build or download a singularity container elsewhere, and then use Allas/SD Connect to import the container to SD Desktop.

If you have a root access to a machine with singularity, you can build your own container, that contains exactly the software and datasets you need. Many software are also available as ready made Singularity containers or as Docker containers that can be converted into Singularity containers. In this document we show, how to import a ready-made Singularity container from a public repository to SD Desktop.

Importing ready made container through Puhti

In the example below we import BETA Binding and Expression Target Analysis software to SD Desktop. This tool is available as a ready made Singularity container in Biocontainers repository. You can find the tool by searching for Binding and Expression Target Analysis in the repository. When you open the detailed information of the resulting cistrome_beta container, you can see that the sigularity module can be downloaded from url: "https://depot.galaxyproject.org/singularity/cistrome_beta:1.0.7--py27heb79e2c_4"

As we don't need to build the container from scratch, we can use puhti.csc.fi server to download the container image and push it to Allas.

First login to puhti.csc.fi. Then, start an interactive batch job session with command:

sinteractive

In the interactive session move to LOCAL_SCRATCH directory an set some Singularity related environment variables:

export SINGULARITY_TMPDIR=$LOCAL_SCRATCH
export SINGULARITY_CACHEDIR=$LOCAL_SCRATCH
unset XDG_RUNTIME_DIR

Then download a local copy of the Beta container with command

singularity pull beta.sif https://depot.galaxyproject.org/singularity/cistrome_beta:1.0.7--py27heb79e2c_4

This creates a new singularity container file, beta.sif, is then uploaded allas that. From the home page of BETA software we download also a test data set for confirming that the container works.

wget http://cistrome.org/BETA/src/BETA_test_data.zip

Then we upload these two files to Allas. In this example we use project 2012345.

module load allas
module load biokit
module load biopythontools
allas-conf project_2012345
a-put --sdx beta.sif -b 2012345_beta
a-put --sdx BETA_test_data.zip -b 2012345_beta

The commands above store the files into bucket 2012345_beta in Allas. a-put is used with option --sdx in order to encrypt the uploaded data with SD Desktop compatible encryption.

Using a container in SD desktop

Once the sif formatted singularity container file and the sample data has been uploaded to Allas, we can copy them to SD Desktop. To do this open Data Gateway, in your session SD Desktop. After that copy the data to local disk in SD Desktop. You can do that using the graphical tools on the Desktop, or by using Linux command line: Open a Linux terminal in the SD-Desktop. In the terminal, move the singularity file and test data to your current locations:

cp Projects/SD\ connect/project_201234/2012345_beta/beta.sif ./
cp Projects/SD\ connect/project_201234/2012345_beta/BETA_test_data.zip ./

Unzip the test dataset:

unzip BETA_test_data.zip

Now you can run BETA through singularity command. For example the help of command BETA minus is shown with command:

singularity exec beta.sif BETA minus -h

And the analysis with sample data in directory BETA_test_data can be executed with commands like:

singularity exec beta.sif BETA minus -p BETA_test_data/3656_peaks.bed --bl -g hg19

In this example the results will be written to directory BETA_OUTPUT.