S3 interface to object storage¶

Instead of Rclone, we can use s3 APIs to connect to object storage. You may use:

s3cmd to manipulate buckets (create, get/put objects) from command line
s3fs to mount containers as filesystems

Both tools need ec2 credentials to be created.

Create EC2 credentials¶

Create and download an application credential from openstack dashboard as app-credentials.sh.

You need to install the Openstack cli as described here in the cli tutorial.

Then execute the content of the file:

$ source app-credentials.sh

And create the ec2 credentials:

$ openstack ec2 credentials create -c access -c secret  -f value | paste -sd: | tee ~/.passwd-s3fs ~/.s3cfg

S3cmd: manipulate object storage from command line¶

This command does not seem to like capital letters neither in bucket names nor in directory names.

Install s3cmd¶

On Ubuntu machine, you can install the packaged version with the following commands:

$ sudo apt update
$ sudo apt install s3cmd

Configure environment¶

Modify ~/.s3cfg:

$ nano `~/.s3cfg

Comment the content adding # before the string:

#<ACCESS-KEY>:<SECRET-KEY>

and add the following content:

[default]
access_key = <put_your_ACCESS-KEY_here>
secret_key = <put_your_SECRET-KEY_here>
host_base = swift.cloud.garr.it
host_bucket = %(bucket).swift.cloud.garr.it
use_https = True

Check everything is correct by executing a simple command, which should return the list of buckets in your object storage area:

$ s3cmd ls
2022-04-21 10:31  s3://fulvio
.....

Use s3cmd¶

Short summary of most common commands, for more information please visit the s3cmd page.

List contents:

$ s3cmd ls
2022-04-21 10:31  s3://fulvio

Create new bucket:

$ s3cmd mb s3://mynewbucket/
Bucket 's3://mynewbucket/' created

Put file:

$ s3cmd put testsmallfile s3://mynewbucket/
upload: 'testsmallfile' -> 's3://mynewbucket/testsmallfile'  [1 of 1]
10485760 of 10485760   100% in    0s    30.37 MB/s  done

Recursive copy, put whole directory (note missing trailing ‘/’):

$ s3cmd put -r testdir s3://mynewbucket/
upload: 'testdir/aRandomFile.png' -> 's3://mynewbucket/testdir/aRandomFile.png'  [1 of 1]
67819 of 67819   100% in    0s  1577.91 kB/s  done

Get a file (destination file name can be omitted, default to same name as remote):

$ s3cmd get s3://mynewbucket/testdir/aRandomFile.png copyOfRandomFile.png
download: 's3://mynewbucket/testdir/aRandomFile.png' -> 'copyOfRandomFile.png'  [1 of 1]
67819 of 67819   100% in    0s     3.05 MB/s  done

Delete a file (enable recursion with ‘-r’):

$ s3cmd del s3://mynewbucket/testdir/aRandomFile.png
delete: 's3://mynewbucket/testdir/aRandomFile.png'

Delete bucket (must be empty):

$ s3cmd rb s3://mynewbucket
Bucket 's3://mynewbucket/' removed

S3fs: mount a container as a filesystem¶

Install s3fs¶

On Ubuntu machine, you can install the packaged version with the following commands:

$ sudo apt update
$ sudo apt install s3fs

Check the version:

$ s3fs --version

N.B. These instructions refer to version 1.86 available on Ubuntu 20.04. Different versions may require different configuration.

Uncomment user_allow_other option by removing the #:

$ nano /etc/fuse.conf

Assign the right permissions to configuration file:

$ chmod 600 ~/.passwd-s3fs

Mount a container¶

First, you need to create the container on your openstack project. Then you can mount your container on your local directory. Assume that you have a container named test_container and a local directory named test_dir:

$ s3fs test_container test_dir -o  allow_other -o host=https://swift.cloud.garr.it -o use_path_request_style -o umask=000

Now your container has been mounted on test_dir directory. You can access it and every change you make inside the directory is istantly made inside the container.

Debug¶

If you need to debug, add the following options at the mounting command:

-o dbglevel=info -f -o curldbg

Note that -f option cause the command to run in foreground, so CTRL+C will kill the command.

Unmount¶

$ umount test_dir