Skip to content

How can I mount my Allas S3 bucket to a VM running in cPouta

Combining cPouta cloud environment and Allas storage environment allows you to build scalable data management environments. This document shows one example how you can combine these two services by mounting a bucket from Allas to an Ubuntu 22.04 (also tested with Ubuntu 20.04 and 18.04) or a Centos7 based virtual machine running in cPouta.

Installing OpenStack, s3cmd and s3fs

In Ubuntu 22.04 LTS (works for Ubuntu 20.04 and 18.04 as well)

  • After launching an Ubuntu based virtual machine in cPouta, open a terminal connection to the VM and update it with the command:

    sudo apt update

Warning

Older versions of Ubuntu will have older and deprecated versions of python. It is recommended to use the latest Ubuntu version available in Pouta.

  • Then install OpenStack client by:

    sudo apt install python3-pip python3-dev python3-setuptools
sudo pip install --upgrade pip
sudo pip install python-openstackclient

    Info

    For Ubuntu 18.04, type those commands:

    sudo apt install python3-pip python3-dev python3-setuptools
sudo pip3 install --upgrade pip
sudo pip install python-openstackclient --ignore-installed PyYAML

    If you omit --ignore-installed PyYAML, you will receive an error message:

    Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
    It should be installed by distutils, so the removal process cannot confirm which files belong to it.

  • Next, install s3cmd and s3fs commands to your VM.

    sudo apt install s3cmd s3fs

In Centos7 (Maintenance Updates EOL 2024-06-30)

  • After launching a Centos7 based virtual machine in cPouta, open a terminal connection to the VM and update it with the command:

    sudo yum update

  • OpenStack and s3cmd can then be installed by:

    sudo yum install python3 python3-devel wget
sudo python3 -m pip install --upgrade pip
sudo python3 -m pip install python-openstackclient
sudo yum install s3cmd

  • s3fs-fuse can be installed using this command:

    sudo yum install s3fs-fuse

Configuring and using Allas

Use s3cmd to read and write files

Once you have openstack, s3cmd and s3fs installed, download and execute the poutaos_configure tool to configure s3cmd so that it uses your cPouta project. You can also use this tool to switch between different Allas projects if you have several of them.

wget https://a3s.fi/tools/poutaos_configure
chmod u+x poutaos_configure
./poutaos_configure

The poutaos_configure will first ask you for your CSC username and password, you can see which is your CSC username in your My CSC profile page, you csan also change your password there. Then it will list your Allas projects and ask you to fill up the project to be used. Finally it will ask you for the chunk size, it is recommended to leave the default.

After this you can use the storage area of your Allas project with s3cmd commands. Now you can see, download and upload files in this bucket with s3cmd.

  • List all your buckets:
$ s3cmd ls s3://
2022-10-17 07:03  s3://data-europe
2020-09-17 11:12  s3://images-sky
2020-11-06 13:56  s3://case_1
  • Let's assume you already have a bucket called case_1 in Allas and that you have some data objects (i.e. files) in this bucket.
$ s3cmd ls s3://case_1
2022-10-17 07:14     67213268  s3://case_1/file1.txt
  • To retrieve the file:
s3cmd get s3://case_1/file1.txt
  • To upload a new file:
s3cmd put file2.txt s3://case_1/

This is the recommended way to use Allas with the S3 protocol from the command line. However, it is also possible to mount the bucket to your VM so that it is shown as "mounted disk". You can use s3fs for that.

Use s3fs to mount a folder into your VM

  1. To do this, create first an empty directory (like os_case_1) to be used as a mount point:

    mkdir os_case_1

    Info

    Any empty directory can be used as a mount point

  2. Create a .passwd-s3fs file in your home directory. The format of the file must be: ACCESS_KEY_ID:SECRET_ACCESS_KEY and have 600 permissions. (Your project must be sourced: source project_xxxxxxx)

    $ openstack ec2 credentials list -f value | grep $OS_PROJECT_ID | tail -1 |\
   awk '{print $1":"$2}' >.passwd-s3fs
Password:
$ chmod 600 .passwd-s3fs

  3. then use the s3fs command to mount the bucket.

    s3fs case_1 os_case_1 -o passwd_file=~/.passwd-s3fs -o url=https://a3s.fi/ \
    -o use_path_request_style -o umask=0333,uid=$(id -u)

    Info

    The uid value returned by id -u should be 1000 for the default user

    Info

    The umask value 0333 mounts the files in read-only mode. If you want to mount them in read-write mode, use 0027 instead

  4. And after this you should be able to see the objects of the mounted bucket as files. Try for example the command:

    ls -l os_case_1

    The output should be the same as with s3cmd ls s3://case_1

    Info

    You can also check the mount by typing the command df -h

  5. When you are done you can unmount the folder by:

    sudo umount os_case_1