Transferring data in and out of ERIS

Posted on
Updated on

December 2, 2024

Files can be copied to and from ERISTwo using Windows network file-sharing to server address erisonefs.partners.org or over secure shell protocol (SFTP or RSYNC) to server address erisonexf.partners.org.  For options to transfer directly from other network file shares see the FAQ on Accessing remote fileshares from the ERISTwo Cluster.

Data transfer speed considerations 

Data transfer speed will be slow when using a wireless network, or the VPN - use a wired ethernet connection within the Mass General Brigham network if possible.  Many wired ethernet connections at Mass General Brigham are limited to 100Mb/s.  if you have a 100Mb/s connection and will be frequently transferring files to ERISTwo, contact helpdesk to request an upgrade of your connection to 1000Mb/s.

Transferring many small files will be many times slower than transferring the same data in a compressed archive such as a "Zip" file. 

External hard drives are slower than the speed of the network and therefore limit the speed of data transfer.  The maximum speed for USB (1) is 11Mb/s while USB2 is 480Mb/s. 

The type of data storage drive in your computer makes a difference - Solid State drives (SSD) can read and write faster than 1Gb/s network transfer rates, but many Hard Disk Drives (HDD) limit data transfer to 320Mb/s or 640Mb/s depending on the age and size of the drive.  Simple applications are available to test the read and write speed of your storage drive.

Mapping a network drive

This is the easiest method available. All home directories are available to map as network drives.  Data directories are available by default, but can be restricted to read-only access at the request of the data owner / PI to Scientific Computing.

Note: Only home folders and BriefCASE data folders can be accessed via CIFS.  A data folder on the mini-BriefCASE class of storage cannot be accessed via CIFS, please use SFTP instead.  

From Microsoft Windows

Type the network path into the address bar of the File Explorer

  • The network path to the home directory for a Mass General Brigham user abc123 would be 
\\erisonefs.partners.org\homes\abc123

or possibly

\\erisonefs.partners.org\abc123
  • The network path to data folder "/data/labName" would be 
\\erisonefs.partners.org\labName

When connecting from a computer that is not a Mass General Brigham workstation, connect with your username, for example "PARTNERS\abc123"

From MacOS

Open the "Finder" application, select "Go" from the application menu, and "Connect to Server".  Enter the server address:

  • The server address path to the home directory for Mass General Brigham user abc123 would be 
smb://erisonefs.partners.org/abc123
  • The server address for data folder "/data/labName" would be 
smb://erisonefs.partners.org/labName

Note: On some MacOS versions, using an address beginning with "smb://" does not work with our server, use "cifs://" instead.

From Linux

From the "Places" menu, select "Connect to Server" and enter the following details (using Mass General Brigham user ID abc123 as an example):

  • Service type: windows share
  • Server: erisonefs.partners.org
  • Share: abc123   (or the name of a lab folder under /data)
  • User: abc123
  • Domain Name: PARTNERS

Using a graphical SFTP application

This is the best option to get started quickly. Cyberduck is a graphical applications that runs on Windows and Mac. Linux has built in support for SSH file transfer in the "Gnome" desktop.  Enter the server name eristwo.partners.org and your account details as shown in the screenshot attached below and press "Quickconnect".  The first time connecting you will be asked to accept the SSH key.  Drag and drop files or folders between the two panels to copy between your PC and the cluster.

Options using the command line

Use erisonexf.partners.org as the server name for SCP/SFTP, with the default Port 22

  • scp is suitable for copying files and folders.
  • sshfs allows you to access to your folder on the cluster as you would access a remotely shared folder

Installing these depends on your platform.

Synchronization options

Use erisonexf.partners.org as the server name for synchronization options, with the default Port 22

  • Unison synchronizes your folder on the cluster with a folder on your workstation each time you run the application. It is essential to install the same version as the cluster, which is 2.27.57. There are both graphical and command line interfaces to unison. We recommend running it regularly to keep a local copy of your cluster data and configuring it to keep one or two backups of each changed file in case you wish to go back to an earlier version. Note that it is easy to install on Linux, moderate on a Mac and difficult on Windows.
  • rsync does one way syncs - run in once to get data on the cluster and run it with destinations reversed to get data off. There are also graphical front ends available
  • BitTorrent Sync has been reported to work

High-speed data transfer

Mapping a network drive is convenient but may not transfer data at the full speed of the network.  First try the network drive option, but contact Scientific Computing for details of high-speed transfer options if this option is not fast enough.

Some transfer methods make better use of the available network bandwidth than others and are therefore faster for transferring large amounts of data.  If "rsync" can be installed on your computer this is generally faster than SCP/SFTP for example.

Screenshot of settings for CyberDuck

 

Go to KB0027929 in the IS Service Desk