Using Allas and Lumi-O from LUMI supercomputer

At the moment the LUMI object storage service, LUMI-O, is not yet in use and tools for using object storage services are not by default installed in LUMI-C. In this document we describe how you can install commonly used object storage clients to LUMI-C and how to configure connection to Allas object storage services. These tools can be used to access Lumi-O too when it becomes available.

Installing Object Storage tools for Allas and Lumi-O

In this example we do the installation to the /project area in Lumi so that all project members can utilize these tools. In this example we use example project 462000007. Please use your own project number when installing these tools for your own use.

Once you have logged in to Lumi, move to the /project disk area and make there a directory called allas.

cd /project/project_462000007
mkdir allas
cd allas

Then, clone allas-cli-utils repository to this location and continue to the cloned directory:

git clone https://github.com/CSCfi/allas-cli-utils  
cd allas-cli-utils

Allas-cli-utils has a list of software dependencies that needs to be be installed. We use here the lumi-container-wrapper tool to install most of them (s3cmd, Python OpenStack clinet, Python swift client, zstdmt and crypt4gh). The installation is done with commands:

module load LUMI lumi-container-wrapper
conda-containerize new --prefix /project/project_462000007/allas allas-dependencies.yaml

The command above will install the dependencies to a singularity container and make command-like execution scripts to a bin directory that locates in the directory defined with the --prefix option. (in this case /project/project_462000007/allas/bin).

Finally we need to modify a_env_conf file a bit so that a-tools works smoothly. Open the file:

nano allas-cli-utils/a_env_conf

In this file do following modifications (remember to use your own project number instead of 462000007)

local_host=”lumi”
allas_conf_path="/project/project_462000007/allas/allas-cli-utils/allas_conf"
tmp_root="/scratch/project_462000007"

Next move to the allas directory and create a set up file allas_setup.sh which will be use to simplify the setup process of Allas tools

cd /project/project_462000007/allas 
nano allas_setup.sh

In the setup file will be used to add the installation directories to a-tools and object storage software your command path . In addition you create alias 'allas-conf' that runs source command for the connection setup script allas_conf. (remember here too to replace the project number with your own project)

export PATH=/project/project_462000007/allas/allas-cli-utils:/project/project_462000007/allas/bin:$PATH
alias allas-conf="source /project/project_462000007/allas/allas-cli-utils/allas_conf"

Now the installation is ready. In the future you and your group members need to only run the setup commands described below, to enable object storage tools and to open connection to Allas and Lumo-O

Using Lumi-O and Allas

Once the object storage tools have been installed to the project directory as described above, then opening connection to Allas or Lumi-O requires first setting up the environment with command:

source /project/project_your-project-number/allas/allas_setup.sh

After this commands like allas-conf, rclone, swift or a-put should work in the same way as in Puhti and Mahti.

Running command allas-conf starts normal configuration process for a swift based connetion to Allas:

allas-conf

If you want to configure connetion Lumi-O, run command:

allas-conf --lumi

This command asks you to connect with your browser to Lumi-O configuration sever, create credentials there and the copy the project nunber and keys for the setup tool. The setup process for Lumi-O will create environment variables needed for S3 command and confuguration files for s3cmd and rclone. In addition you can define that a-commands will use by default Lumi-O storage server in stead of Allas. After that commands like a-list, a-put or a-get will use your Lumi-O storage. If you don't set Lumi-O as the default storage serice, you can add option --lumi to a-commands to use Lumi-O instead of Allas.

For rclone, Lumi-o configuration provides two rclone remotes: lumi-o: and lumi-pub: . The buckets used by lumi-pub will be publicly visible in URL: https://project-number.lumidata.eu/bucket_name.

Note, that you can have active connection to both Lumi-O and Allas in the same time.

For example, if you would first open Allas connection with command:

source /project/project_your-project-number/allas/allas_setup.sh
allas-conf

And then open Lumi-O connection with:

allas-conf --lumi

(when running the latter command we accept that Lumi-O will be the default erver for a-commands) Now you can list the buckets available in Lumi-O with commands:

a-list

or

rclone lsd lumi-o:

And in the same time you can list your buckets in Allas with commands:

a-list --allas

or

rclone lsd allas:

Copying data from Allas to Lumi-O could now be done with command:

rclone copyto -P allas:bucket-in-allas/object lumi-o:bucket-in-lumi-o/object

The command above will work only for files smaller than 5 GB.