Transferring data with Rclone

Rclone is a powerful command line program for data transfer with high performance. It was designed for managing data in cloud storage, but also works just as well for data transfer to/from ERIS cluster storage. You can find instruction for download and installation of rclone on their website. Here we will show how to configure rclone to transfer data to ERIS cluster storage.

1. Use rclone to transfer data to ERIS cluster storage.

After installing rclone on your local desktop/laptop you will need to configure a new remote connection with

rclone config

and from the startup menu choose New remote by entering n in the prompt. After that you will rclone will prompt you a number of configuration options to setup a new remote connection. Here are the options you will need to specify:

  1. Name for new remote - you can pick any name, here we will use erisstorage as an example.
  2. Type of storage to configure - SSH/SFTP,  you can enter number 38 to make this selection.
  3. SSH host to connect to - erisonexf.partners.org.
  4. SSH username - enter your username on ERIS cluster.
  5. SSH port number - press Enter to select the default value (port 22).
  6. SSH password - you can leave this blank or enter y to allow rclone to store you password (it is encrypted).  

Rclone will prompt you to other options during this configuration process, but for all the other options you can pick the default values and simply press Enter when prompted. 

After remote connection is configured you can transfer data to ERIS cluster storage with

rclone copy files_to_transfer erisstorage:/path/to/destination

and transfer data from ERIS cluster storage to your local machine with

rclone copy erisstorage:/path/to/files/to/transfer /path/to/destination

2. Use rclone to transfer data from ERIS cluster storage to another remote server.

Rclone is installed on ErisTwo and it is available with

module load rclone

You will need to configure a new rclone remote connection using

rclone config

and following the same instructions as above. The only difference is in step 3 you will need to use the hostname of the server where you want to transfer the data.