Rclone quick tutorial: interaction with object storage¶

Installing and configuring¶

Linux:

If not yet installed on your machine, you can install rclone with the following command:

$ curl https://rclone.org/install.sh | sudo bash

Windows:

In order to use Rclone on Windows systems, you need a bash. If you don’t have one yet, you can download git bash from:

https://gitforwindows.org/

Install it and then download rclone from:

https://rclone.org/downloads/

Unzip the archive and open the bash in the rclone directory. From here, you can use rclone command as follow:

./rclone

Download OpenStack credentials¶

Create and download an application credential from openstack dashboard as app-credentials.sh.

In order to use Rclone, you can either load the configuration from environment either write them directly in the configuration file. Choose the best approach for your needs.

In both cases, you should edit the Rclone configuration file:

$ nano .rclone.conf

Case 1: Take variables from environment¶

Add the following text to rclone.conf:

[garr-cloud]
type = swift
env_auth = true

Then execute the content of the file:

$ source app-credentials.sh

Case 2: Write variables in the configuration file¶

Take note of these three variables in app-credentials.sh:

OS_REGION_NAME
OS_APPLICATION_CREDENTIAL_ID
OS_APPLICATION_CREDENTIAL_SECRET

Add the following text to rclone.conf:

[garr-cloud]
type = swift
auth = https://keystone.cloud.garr.it:5000/v3/
auth_version = 3
region = <insert here the content of OS_REGION_NAME>
application_credential_id = <insert here the content of OS_APPLICATION_CREDENTIAL_ID>
application_credential_secret = <insert here the content of OS_APPLICATION_CREDENTIAL_SECRET>

Mind that env_auth = true takes variables from environment, so you shouldn’t insert it in this case.

Case 3: Use EC2 credentials¶

First, you need to install the Openstack cli as described here in the cli tutorial.

Then execute the content of the file:

$ source app-credentials.sh

And create the ec2 credentials:

$ openstack ec2 credentials create

Take note of these two variables:

+------------+-----------------------+
| Field      | Value                 |
+------------+-----------------------+
| access     | <access_key>          |
| secret     | <secret_key>          |
+------------+-----------------------+

Add the following text to rclone.conf:

[garr-cloud]
type = s3
provider = AWS
access_key_id =  <insert here the content of access_key>
secret_access_key =  <insert here the content of secret_key>
endpoint = https://swift.cloud.garr.it

Note

You can use EC2 credentials to access object storage with other tools. Check S3 interface to object storage.

Check configuration¶

Note

On windows systems you have to run rclone inside the executable folder and prepend ./ (i.e. ./rclone)

Now you can verify your configuration with:

$ rclone lsd garr-cloud:
$ echo $?

The command above will most probably return no output, but the exit code should be ‘zero’.

Check total used space¶

Unfortunately, at time of writing rclone capability of interfacing with the Ceph storage system is rather limited, and the command does not provide information on quota limits. However, it does provide information on the total used space:

$ rclone about garr-cloud:
Used:    190.364M
Objects: 19

Copy files and directories to cloud¶

Make a container on your remote object store:

$ rclone mkdir garr-cloud:test-cont

Copy a local file to the container:

$ rclone copy sample_file.txt garr-cloud:test-cont
$ rclone ls garr-cloud:test-cont
    1103 sample_file.txt

Now, suppose you have these files on your local filesystem:

$ ls -lR /tmp/test_dir/
    /tmp/test_dir/:
    total 8
    -rw-rw-r-- 1 ubuntu ubuntu 1103 Nov 13 15:31 file1.txt
    drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 13 15:32 subdir1

    /tmp/test_dir/subdir1:
    total 4
    -rw-rw-r-- 1 ubuntu ubuntu 459 Nov 13 15:32 file2.txt

Execute the following command to synchronize it with the remote:

$ rclone sync /tmp/test_dir/ garr-cloud:test-cont/sublevel
$ rclone ls garr-cloud:test-cont
    1103 sample_file.txt
    1103 sublevel/file1.txt
    459 sublevel/subdir1/file2.txt

Mind behaviour of sync! It makes destination identical to source

Copy files from remote to local¶

The following command copies files from remote to a local directory, create it if not exists:

$ rclone -P copy garr-cloud:test-cont/sublevel checkDir/
    Transferred:        1.525k / 1.525 kBytes, 100%, 4.222 kBytes/s, ETA 0s
    Errors:                 0
    Checks:                 0 / 0, -
    Transferred:            2 / 2, 100%
    Elapsed time:       300ms

$ ls -lR checkDir/
    checkDir/:
    total 12
    -rw-rw-r-- 1 ubuntu ubuntu 1103 Nov 13 15:31 file1.txt
    -rw-rw-r-- 1 ubuntu ubuntu 1103 Nov 13 15:28 sample_file.txt
    drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 13 15:53 subdir1

    checkDir/subdir1:
    total 4
    -rw-rw-r-- 1 ubuntu ubuntu 459 Nov 13 15:32 file2.txt

Mounting object storage on local filesystem¶

Linux:

First, you need to create a directory on which you will mount your filesystem:

$ mkdir ~/mnt-rclone

Then you can simply mount your object storage with:

$ rclone -vv --vfs-cache-mode writes mount garr-cloud: ~/mnt-rclone

Windows:

First you have to download Winsfp:

http://www.secfs.net/winfsp/rel/

WinFsp is an open source Windows File System Proxy which provides a FUSE emulation layer.

Then you can simply mount your object storage with (no need to create the directory in advance):

./rclone -vv --vfs-cache-mode writes mount garr-cloud: C:/mnt-rclone

vfs-cache-mode flag enable file caching, you can use either writes or full option. For further explanation you can see official documentation at the link: https://rclone.org/commands/rclone_mount/#file-caching

Now that your object storage is mounted, you can list, create and delete files in it.

Unmount object storage¶

To unmount, simply press CTRL-C and the mount will be interrupted.

Create encrypted directory¶

You can encrypt a directory in your remote container and decrypt it easily through rclone commands. First, we create a new remote in the rclone configuration file. It will be a subdirectory of your working remote (i.e. garr-cloud remote and crypt-dir directory). So, everything inside garr-cloud:crypt-dir will be encrypted and anything outside won’t.:

$ nano .rclone.conf

Copy and paste the following text at the end of the file:

[garr-cloud-crypt]
type = crypt
remote = garr-cloud:crypt_dir
filename_encryption = standard
directory_name_encryption = true

Then set the passwords that will be saved obscured inside the config file:

$ rclone config password garr-cloud-crypt password <type_a_password>
$ rclone config password garr-cloud-crypt password2 <type_another_password>

Now we need to create crypt_dir directory inside garr-cloud:

$ rclone mkdir garr-dev:crypt_dir

or, if you have garr-cloud still mounted on ~/mnt-rclone:

$ mkdir ~/mnt-rclone/crypt_dir

Every file you will create inside garr-cloud-crypt container will be encrypted. You can try it following these steps. First, we create a file to copy:

$ echo "Hello world" > test.txt

Then, we copy it to the remote, in a new directory named test_dir:

$ rclone copy test.txt garr-cloud-crypt:test_dir

Now we can list the new created files through the mounted filesystem:

$ cd ~/mnt-rclone/crypt_dir/
$ ls -R

You will get an output similar to this:

.:
0ruoo4gjnnuk01p4gok56li8ts

./0ruoo4gjnnuk01p4gok56li8ts:
b4tc2rcdasquuns71k9fa2uiss

and if you cat the content of the file, you will see that it has been encrypted:

$ cat 0ruoo4gjnnuk01p4gok56li8ts/b4tc2rcdasquuns71k9fa2uiss

To access and decrypt the file in a complete transparent way, you can copy it from the remote through:

$ rclone ls garr-cloud-crypt:

    1048576 test_dir/test.txt

$ rclone copy garr-cloud-crypt: new_dir
$ ls new_dir

    new_dir/:
    test_dir

    new_dir/test_dir:
    test.txt

$ cat new_dir/test_dir/test.txt
    Hello world