Tykky is a set of tools which make software installations to HPC systems easier and more efficient using Apptainer containers.
Tykky use cases:
- Conda installations, based on Conda
- Pip installations, based on pip
- Container installations, based on existing Docker or Apptainer/Singularity images.
- This includes installations from the Bioconda channel, see this tutorial for an example.
Tykky wraps installations inside an Apptainer/Singularity container to improve startup times, reduce IO load, and lessen the number of files on large parallel filesystems. Additionally, Tykky will generate wrappers so that installed software can be used (almost) as if it was not containerized. Depending on tool selection and settings, either the whole host filesystem or a limited subset is visible during execution and installation. This means that it's possible to wrap installation using e.g mpi4py relying on the host provided mpi installation.
This documentation covers a subset of the functionality and focuses on conda and Python, a large part of the advanced use-cases are not covered here yet.
As Tykky is still under development some of the more advanced features might change in exact usage and API.
To access Tykky tools:
1) Usually it is best to first unload all other modules:
2) Load Tykky module.
module load tykky
Conda based installation
First make sure that you have read and understood the license terms for miniconda and any used channels before using the command.
- Miniconda end user license agreement.
- Anaconda terms of service.
- A blog entry on Anaconda commercial edition.
1) Create conda environment file env.yml:
- Create manually a new file or
- Create the file from existing conda installation. For example:
conda env export -n <target_env_name> > env.yml.
An example of a suitable
env.yml file would be:
channels: - conda-forge dependencies: - python=3.8.8 - scipy - nglview
2) Create new directory for installation
/projappl/<your_project>/.. is a good place.
3) Create installation
conda-containerize new --prefix <install_dir> env.yml
4) Add the bin directory
<install_dir>/bin to the path.
5) You can call python and any other executables conda has installed in the same way as if you had activated the environment.
pip with conda
To install some additional pip packages, add the
-r <req_file> argument e.g:
conda-containerize new -r req.txt --prefix <install_dir> env.yml
The tool also supports using mamba
for installing packages. Mamba often finds suitable packages much faster than conda, so it is a good option when required package list is long. Enable this feature by adding the
conda-containerize new --mamba --prefix <install_dir> env.yml
Create new conda based installation using the previous
mkdir MyEnv conda-containerize new --prefix MyEnv env.yml
$ export PATH="$PWD/MyEnv/bin:$PATH" $ python --version 3.8.8 $ python3 Python 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import scipy >>> import nglview >>>
Modifying a conda installation
Tykky installed software resides in a container, so it can not be directly modified.
Small Python packages can be added normally using
pip, but then the Python packages are
sitting on the parallel filesystem so this is not recommended for any larger installations.
To actually modify the installation we can use the
together with the
--post-install <file> option which specifies a bash script
with commands to run to update the installation. The commands are executed
with the conda environment activated.
conda-containerize update <existing installation> --post-install <file>
<file> could e.g contain:
conda install -y numpy conda remove -y nglview pip install requests
In this mode the whole host system is available including all software and modules.
Pip based installations
Sometimes you don't need a full blown conda environment or you might prefer pip to manage Python installations. For this case we can use:
pip-containerize new --prefix <install_dir> req.txt
req.txtis a standard pip requirements file. The notes and options for modifying a conda installation apply here as well.
Note that the Python version used by
pip-containerize is the first Python executable found in the path, so it's affected by loaded modules.
Important: This python can not be itself container-based as nesting is not possible.
An additional flag
--slim argument exists, which will instead use a pre-built minimal python
container with a much newer version of python as a base. Without the
--slim flag, the whole host system is available,
and with the flag the system installations (i.e /usr, /lib64 ...) are no longer taken from the host, instead
coming from within container.
Container based installations
Tykky also provides an option to:
- Generate wrappers for tools in existing Apptainer/Singularity containers, so that they can be used
transparently (no need to prepend
singularity exec ..., or modify scripts if switching between containerized versions and "normal" installation).
- Install tools available in Docker images, including generating wrappers.
wrap-container -w </path/inside/container> <container> --prefix <install_dir>
<container>can be a local filepath or any URL accepted by singularity (e.g
-wneeds to be an absolute path (or comma separated list) inside the container. Wrappers will then be automatically created for the executables in the target directories / for the target path. If you do not know the path of executables in the container, open a shell inside the container and use which command. To open shell:
- In case of existing local Apptainer/Singularity file:
singularity shell xxx.sif.
- In case of Docker or non-local Apptainer/Singularity file, create first the installation with some path and then start with created
- In case of existing local Apptainer/Singularity file:
More complicated example
How it works
See the README in the source code repository. The source code can be found in the GitHub repository.