Building Apptainer containers from scratch
On CSC platforms, there are two ways to use Apptainer containers (formerly known as Singularity) that you have built yourself. You may:
- Convert Docker containers to Apptainer image files, or
- Build an Apptainer image entirely from scratch.
This tutorial focuses on the latter scenario, with further information available on the Apptainer website.
Containers as distinct environments
One can think of containers as distinct environments of their own. For example, the Linux distribution used in the container may differ from that of the host environment where the container is executed. Apptainer images also have an internal file system that is not visible outside the container. Likewise, most of the host environment will not be visible to a container unless explicitly specified. Selected folders can be exposed to the container by using bind mounts.
When planning the contents and configuration of an Apptainer container, a decision needs to be made on whether specific installations should be done within the container or added as bind mounts. For example, certain dependencies may already be present on the host. The container Linux distribution will naturally affect this decision.
Frequently the environment variables used within a container versus the host environment will also differ (more on that below).
Preparing a definition file
A definition file (with the extension .def) is a file containing the commands required to build the container. Examples of definition files used in CSC's computing environment can be found in our singularity-recipes repository.
A definition file is split into sections serving different purposes (see here for details). At the very top of the file, one must provide a header specifying a bootstrap keyword and a Linux distribution. For example, to create a header for a CentOS 7.7 container, one would type:
Bootstrap: library
From: centos:7.7
The choice of Linux distribution defines the types of commands used in the definition file. For example, a CentOS container would use commands such as yum -y install, whereas an Ubuntu container would use commands such as apt-get -y install.
Other sections one can use include the following. Note that different sections can be added or left out, depending on your needs (e.g. if you have no external files to include in the container, there is no need for a %files section).
%labels(e.g. name and contact of the image maintainer)%files(commands for copying external files into a specific location in the container)%post(section for software installation commands)%environment(environment variables defined upon launching the container)
A few words on environment variables
Environment variables can be defined in both the %post and %environment sections. Those specified in the former may be useful during software installations, but unlike variables specified under the %environment section, they will not be visible upon launching the container.
If you wish to specify environment variables on the host environment in a way that is visible to the container, these must be preceded by APPTAINERENV_. Variables without this prefix will not be passed onto the container.
Building a container
Building an Apptainer container requires root (sudo) access. However, on CSC supercomputers, where users do not have root privileges, you may use the fakeroot feature to circumvent this requirement (with a few restrictions). See our documentation on creating containers for more details.
Alternatively, you may use your own computer, or a virtual computer where you have root access, to build a container image. One option is to use a Pouta virtual machine. This enables one to specify the type of environment used to build the image. Most of the time, the Linux distribution of the environment used to build the container will matter relatively little. However, certain installations (e.g. NVIDIA CUDA drivers) have strict version requirements with regard to kernel headers, meaning that the distribution must effectively be matched between the environment for container building, the container and the host.
To build a container using a definition file, one can simply run:
sudo apptainer build container.sif recipe.def
or without sudo using the --fakeroot option:
apptainer build --fakeroot container.sif recipe.def
Note
In older versions of Linux the command may be singularity instead of apptainer.
If everything is successful, this results in a container image (.sif file) that can be moved to your computing environment of choice. The image file is immutable, meaning that it can not be modified after the build step. It is also possible to create writable sandbox directories that can be used for testing (and can be subsequently converted into Apptainer images).
Running a container
There are a couple of ways to run Apptainer containers in CSC's computing environment. For detailed instructions, visit our documentation on running containers.
The options include using:
apptainer execapptainer_wrapper exec(wrapper script binding local folders, e.g.$TMPDIR)
Many of the containers available in CSC's computing environment come with wrappers for selected commands, enabling them to be executed without a need for the above. It is also possible to prepare wrappers for containers built from scratch, although this requires some familiarity with writing bash scripts. We do also provide a container wrapper tool "Tykky" that can be used to easily containerize and create wrappers for Python environments.
Moving containers to CSC's computing environment
Puhti, Mahti and LUMI
Apptainer image files can be transferred to Puhti, Mahti or LUMI using scp, rsync or a graphical file transfer tool.
CSC Sensitive Data Desktop
If wishing to use a custom-built container on the CSC Sensitive Data Desktop, first transfer the image file to Puhti. Then, follow our guidelines on using Allas to move containers to the SD Desktop.