Extending SD Desktop software environment with your own Apptainer containers
In this tutorial we use cPouta to create a Apptainer containers to import new software to SD Desktop.
Steps 1 ja 2 describe how to setup you own Virtual Machine with singularity environment to cPouta. This is not the only option and if you have already a Apptainer installed elsewhere, you can skip these steps and use your own Apptainer environment.
Step 3 describes one approach to how you can build your own software containers.
Step 4 shows how a container is uploaded to Allas.
Step 5 describes how the software installed in a container can be usen in SD Desktop.
1. Creating your own singularity workbench to cPouta.
In order to utilize all features of Apptainer you must run it in an environment where you have administrator level access rights. At CSC, you can have administrator level access in virtual machines running in cPouta. Using cPouta for the building process adds a bit of extra steps into the process: you have to know how to launch and access virtual machines in cPouta. On the other hand cPouta has a fast connection to the Allas service that is used to import the ready made containers to SD Desktop.
As a first step launch virtual machine in cPouta as described in'cPouta user guide * Launching a virtual machine with the cPouta web interface
and for example these tutorial videos:
- Launching a virtual machine and connecting to it on a local macOS laptop
- Creating a virtual machine in cPouta a webinar recording.
In this tutorial we use a virtual machine that was launched using: * Flavor: Standard.medium * Instance Boot Source: Image * Image Name: Ubuntu-22.04
2. Installing singularity and Allas tools to Ubuntu 22.04 server
Here we start from a situation where we have logged in to our freshly started virtual machine for the first time. As preparatory steps we need to install to our virtual machine Singularity to create new software containers and allas tools to upload the containers we will create to Allas.
The singularity installation is done with commands:
sudo apt update
sudo apt install -y software-properties-common
sudo apt-get install apt-utils
sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y apptainer
After which Allas tools can be installed with:
sudo apt install python3-pip python3-dev
sudo apt-get install python3-setuptools
sudo pip3 install python-openstackclient
sudo apt install python3-swiftclient
curl https://rclone.org/install.sh | sudo bash
git clone https://github.com/CSCfi/allas-cli-utils
3. Creating a Apptainer container
There are many ways to create new Apptainer containers. You can crate the a container by creating a sandbox in to which you log in and add content by typing installation commands. Alternatively you can automatize the installation process so that you collect all the commands and settings to a singularity definition file that instructs the installation process. A detailed view to the container building can be four from the Apptainer user guide.
Here we use a mixture of these two approaches. We first use a simple definition file to create a new container sandbox that contains a set of tools for software installation. Then we open a shell session to the container sandbox and do the actual software installations manually.
3.1 Defaults file
First open a new file called ubuntu_with_inst_tools.def with command:
nano ubuntu_with_inst_tools.def
And copy-paste to the new file the content from the sample definition file below:
Bootstrap: docker
From: ubuntu:20.04
Stage: build
%environment
export TZ=Europe/Helsinki
export LC_ALL=C
export LC_NUMERIC=en_GB.UTF-8
export PATH="/opt/miniconda/bin:$PATH"
%help
Container based on unbuntu containing miniconda.
%runscript
# sample runscript: bamtools passing all arguments from cli: $@
# exec /opt/miniconda/bin/bamtools "$@"
%post
#commands to help installation processes
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
apt update
apt install -y wget bzip2 git autoconf automake build-essential
apt install -y zlib1g-dev pkg-config nano
#install conda
cd /opt
rm -fr miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p /opt/miniconda
mkdir -p /opt/tools
export PATH="/opt/tools/bin:/opt/miniconda/bin:$PATH"
conda config --add channels bioconda
conda config --add channels conda-forge
conda config --set channel_priority strict
3.2 Creating and using sandbox
Next we will use this definition file to create a new Appltainer sandbox
sudo apptainer build --sandbox sd_sandbox_1 ubuntu_with_inst_tools.def
sudo apptainer shell -w sd_sandbox_1
conda init bash
bash
conda create -n biotools
conda activate biotools
conda install bamtools
cd /opt/tools
git clone https://github.com/vcftools/vcftools
cd vcftools/
autoreconf -i
./configure --prefix=/opt/tools
make
make install
New version of samtools could be installed wth command:
apt install samtools
``
Pip can be used to add Python modules the to the Python inside the module:
```text
pip install pyhdfe
When you are ready with the software installations you can exit the sandbox with command:
exit
3.3 Creating a Apptainer image file
Now we are back in the base virtual machine. Next we convert the sandbox into a Apptainer image file with command:
sudo apptainer build sd_tools_1.sif sd_sandbox_1
ls -lh) shows that the current directory has a sandbox directory and an apptainer image file
drwxr-xr-x. 18 root root 4.0K Sep 27 12:56 sd_sandbox_1
-rwxr-xr-x 1 ubuntu ubuntu 419M Sep 27 13:43 sd_tools_1.sif
singularity exec sd_sandbox_1 samtools
singularity exec sd_tools_1.sif samtools
4. Uploading container to Allas/SD Connect
In order to use the Apptainer container in SD desktop, we need to encrypt it with CSC public key and upload it to Allas. If you want to use the same contrainer in other locations too, for example in Puhti and Mahti, you will need to upload another, non-encrypted version to Allas.
For the upload process we use the Allas tools we installed in step 2, were we installed Allas tools to directory $HOME/allas-cli-utils. First we add this directory to command path:
export PATH=${HOME}/allas-cli-utils:${PATH}
source ${HOME}/allas-cli-utils/allas_conf -u kkayttaj
Now we can access Allas with a-tools or rclone. Next we upload the container image we just created to Allas with command:
a-put --sdx sd_tools_1.sif -b 2000123_apptainer_sd -m "SD Compatible. Contains bamtools, samtools and vcftools."
5. Using singularity containers in SD Desktop
In order to use the singularity container you have created you need first download a copy of the container to the SD Desktop with Data Gateway tool. First login to SD Desktop and connect to the Virtual Desktop that you want to use. Open Data Gateway, navigate the to the right project (project_2000123) and bucket (2000123_singularity_sd), and download the singularity image file (sd_tools_1.sif) to the SD Desktop.
After that, open a Linux terminal in the SD Desktop. In the terminal, move the singularity file to the location you want to use it. In this example that could be done with command:
cp /home/kkayttaj/Projects/SD\ connect/project_2000123/2000123_sigularity_sd/sd_tools_1.sif ./
singularity exec sd_tools_1.sif samtools
samtools --help
By default Apptainer bind mounts home directory (/home/$USER), /tmp, and current working directory ($PWD) into your container at runtime. If you need to mount additional directories you will need to define them with singularity option -B source-directory:target-directory.
For example if we have input file input_bam.bam current working directory, it is automatically accessibe for a command that is executed inside the container. But if we need also another input file reference.bed which locates in directory called /data, then we should add that directory to the list if bind mounted directories. For example:
apptainer -B /data:/data exec sd_tools_1.sif samtools depth -a -b /data/refrence.bed input_bam.bam > result.depth