Skip to content

Local Rclone configuration for Allas

Rclone can be installed in any operating system and thus it provides an effective way to use Allas from any computer. You can download a copy of Rclone to your own computer form the link below.

Once you have Rclone installed you still need to configure the connection to Allas. Rclone is a command line application so both configurarion and actual usage is normally done in a command line: Terminal in Mac and Linux, Command prompt or Powershell in Windows.

Configuring Allas connection in Mac and Linux (Swift and S3)

If you are using Rclone in a local Linux or Mac machine, you can download the allas_conf script to set up the connection to your Allas project.

wget https://raw.githubusercontent.com/CSCfi/allas-cli-utils/master/allas_conf
source allas_conf -u your-csc-username -p your-csc-project-name
Note that you should use the -u option to define your CSC username and -p to define the CSC project you want to use. For example:

source allas_conf -u kkayttaj -p project_2001234
The command above defines a connection that uses Swift protocol and is valid in the current terminal session for 8 hours. The Rclone remote site name is in this case allas. After configuration you can for example list your buckets in Allas with command: 
rclone lsd allas:
If you want to use S3 protocol instead of Swift, add option -m S3 to the configuration command. 
source allas_conf -u kkayttaj -p project_2001234 -m S3
Note that in order to run the command above you must have OpenStack client installed in your local machgine. After configuring S3 connection to Allas, you can use it through Rclone remote called s3allas: . For example:

rclone lsd s3allas:
S3 connection is active until you explicitly delete it with command:

source allas_conf -u csc-user-name -p csc-project-name --s3remove

Note

Remember to be careful and security aware, when configuring S3 connection to Allas. The S3 keys are stored in a readble format in you home directory and anyone how can read your keys can access Allas until the keys are expilisitly revoked from Allas. Removing the keys from your own computer is not enough to deactivate them.

Configuring Allas connection in Windows

In Windows machines you can't use allas_conf script. So for Windows some connection specific variables need to be checked up in parallel from a Linux/Mac supporting allas_conf script. Easiest could be Puhti, if you use it otherwise. One option is also to use Windows Linux Subsystem and then the instructions above can be followed.

Configure the Allas connetion with command:

rclone config
You can use this command also in other machines where allas_conf is not available. The command above launches a configration process that you can use to define new Rclone connetion. In Rclone these defined connections are called as remotes.

Below we describe how to create Swift and S3 connections to Allas.

Configuring Swift connection in Windows

Start the process by executing command:

rclone config
And do following selections:

  1. Select n to create a New remote
  2. Name the remote as: allas
  3. From the list of storage protocols, select the number that defines: OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
  4. Select authentication option 2 Get swift credentials from environment vars.
  5. After that select the default blank setting for all the remaining settings until you are back in the starting menu of the confguration process.
  6. Finally, choose q to stop the configuration process.

In the case of Swift you need to do this configuration only once. In the configuration it is now defined that in case of allas, all data for the connection is red from environment variables.

If you have access to Puhti, then the easiest way to check the values of the variables needed is to open terminal connection to it and activate there connection to the Allas project you wish to use. Another option is to use a utility program allas-get-swift-token-win.zip. If you choose to use the utility program, you (or your local it-support) may have to configure your anti-virus etc software to allow running it.

This Swift access is done in Puhti with commands:

module load allas
allas-conf
After that you can check the variable values with commands:

echo $variable_name
The variables required for the Allas Swift access are:

  • OS_STORAGE_URL, Check this value in Puhti with commad echo $OS_STORAGE_URL .
  • OS_AUTH_TOKEN, Check this value in Puhti with commad echo $OS_AUTH_TOKEN .

The value in the variable OS_STORAGE_URL stays the same as long as you are accessing the same project. OS_AUTH_TOKEN variable contains the actual authentication token. This token is valid only for 8 hours so you typically need to generate and check it each time you start using Rclone in your computer.

At the moment we don't have a tool for generating this value in Windows so you need to use Puhti (or some other machine that can run allas_conf tool) to generate this project specific temporary token.

For example, if in Puhti running the allas-conf would produce following values:

module load allas
allas-conf
echo $OS_STORAGE_URL
https://a3s.fi:443/swift/v1/AUTH_5d66718fa0ff46ee1b2581b2225ee1a9d
echo $OS_AUTH_TOKEN
gAAAAABiVAJ08lSVPXH4MqXDb-Vo0KdQgm9reOA7IRf-PavF-RktYpp3K_7kp-5Ck137hsBQl66qRFmd4PE4hEVz0TWwTTmVIPUmmxQ

Then in Windows PowerShell you could set up Allas connection with commands:

$Env:OS_STORAGE_URL = "https://a3s.fi:443/swift/v1/AUTH_5d66718fa0ff46ee1b2581b2225ee1a9d"
$Env:OS_AUTH_TOKEN = "gAAAAABiVAJ08lSVPXH4MqXDb-Vo0KdQgm9reOA7IRf-PavF-RktYpp3K_7kp-5Ck137hsBQl66qRFmd4PE4hEVz0TWwTTmVIPUmmxQ"

In the Windows Command Prompt the corresponding set up commands would be: 

set OS_STORAGE_URL=https://a3s.fi:443/swift/v1/AUTH_5d66718fa0ff46ee1b2581b2225ee1a9d
set OS_AUTH_TOKEN=gAAAAABiVAJ08lSVPXH4MqXDb-Vo0KdQgm9reOA7IRf-PavF-RktYpp3K_7kp-5Ck137hsBQl66qRFmd4PE4hEVz0TWwTTmVIPUmmxQ

With these settings done, you should be able to check your buckets with command: 

rclone.exe lsd allas:

Configuring S3 connection in Windows

Remember that you need to be careful and security aware, when configuring S3 connection to Allas. The S3 keys are stored in a readble format in you home directory and anyone how can read your keys can access Allas until the keys are expilisitly revoked from Allas. Removing the keys from your own computer is not enough to deactivate them.

To check your S3 keys, open connection to Puhti (or some other machine that can run allas_conf tool). If you don't yet have S3 connetion to your Allas project activated, open it with commands:

module load allas
allas-conf -m S3
This creates a local configuration files that we will later on use to check some values.

Then switch to you Windows Command Prompt and start the rclone configuration process by executing command:

rclone config
And do following selections:

  1. Select n to create a New remote
  2. Name the remote as: s3allas
  3. From the list of storage protocols, select the number that defines: Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, and Tencent COS
  4. Next you are asked to Choose your S3 provider. Select option that provides Any other S3 compatible provider
  5. The select that you Enter AWS credentials in the next step
  6. Give the AWS access key. You can check this in Puhti with command: grep access_key $HOME/.s3cfg | cut -d " " -f3
  7. Give the AWS secret access key. You can check this in Puhti with command: grep secret_key $HOME/.s3cfg | cut -d " " -f3
  8. Region 1
  9. Endpoint: a3s.fi
  10. Location constant: (blank, just press entwer)
  11. acl: 1
  12. Edit advanced config: n
  13. Remote config: y Yes, this is ok.
  14. q, quit config

After this you can access your Allas project using Rclone remote s3allas. For example:

rclone.exe lsd s3allas:

This connection stays active in your laptop until you delete it.