Transferring data in and out of ERISTWOFS

Users can map their ERISTwo home directories and briefcases to their PC desktops by accessing the file-sharing server eristwofs.partners.org. By doing so users can conveniently transfer files between their PC and the ERISTwo cluster. For Windows and Mac this process uses the smb protocol while for linux it is cifs. Examples of implementation will be given below.

We note that files can also be transferred to/from eristwofs.partners.org using SFTP (secure file transfer over SSH) and where an example of how to do this using an sftp client tool such as CyberDuck is illustrated below. This approach is to be preferred for large (>100GB) file sizes.

Data transfer speed considerations

Data transfer speed will be slow when using a wireless network, or the VPN - use a wired ethernet connection within the Mass General Brigham network if possible.  Many wired ethernet connections at Mass General Brigham are limited to 100Mb/s.  if you have a 100Mb/s connection and will be frequently transferring files to ERISTwo, contact helpdesk to request an upgrade of your connection to 1000Mb/s.

Transferring many small files is much slower than transferring the same data in a compressed archive such as a "Zip" file. 

External hard drives are slower than the speed of the network and therefore limit the speed of data transfer.  The maximum speed for USB (1) is 11Mb/s while USB2 is 480Mb/s. 

The type of data storage drive in your computer makes a difference - Solid State drives (SSD) can read and write faster than 1Gb/s network transfer rates, but many Hard Disk Drives (HDD) limit data transfer to 320Mb/s or 640Mb/s depending on the age and size of the drive.  Simple applications are available to test the read and write speed of your storage drive.

Mapping a network drive

All home and briefcase (group storage) directories are available to be mapped as network drives. If they so wish, the PI or data owner for a particular briefcase can submit a request to @email to restrict the mapped drive to read-only access.

From Microsoft Windows

Type the network path into the address bar of the File Explorer

  • The network path to the home directory for a Mass General Brigham user abc123 would be 
\\eristwofs.partners.org\homes\abc123

or possibly

\\eristwofs.partners.org\abc123
  • The network path to data folder "/data/labName" would be 
\\eristwofs.partners.org\labName

When connecting from a computer that is not a Mass General Brigham workstation, connect with your username, for example "PARTNERS\abc123"

From Mac OS X

Open the "Finder" application, select "Go" from the application menu, and "Connect to Server".  Enter the server address:

  • The server address path to the home directory for Mass General Brigham user abc123 would be 
smb://eristwofs.partners.org/abc123
  • The server address for data folder "/data/labName" would be 
smb://eristwofs.partners.org/labName

Note: On some Mac OS X versions, using an address beginning with "smb://" does not work with our server, use "cifs://" instead.

From Linux

From the "Places" menu, select "Connect to Server" and enter the following details (using Mass General Brigham user ID abc123 as an example):

  • Service type: windows share
  • Server: eristwofs.partners.org
  • Share: abc123   (or the name of a lab folder under /data)
  • User: abc123
  • Domain Name: PARTNERS

Using SSH/SFTP

GUI client tool

A convenient Graphical User Interface or GUI tool is Cyberduck which is available for both Mac and Windows platforms. For Windows users there is also the popular utliity WinSCP. On Linux platforms the OS will often provide a GUI sftp client tool as is found in "Gnome" desktop for example.

For configurring CyberDuck, enter the server name eristwofs.partners.org and your account details as shown in the screenshot below and press "Quickconnect". 

When connecting for the first time you will be asked to accept the SSH key. Subsequently, drag and drop files or compressed (i.e. zipped or tar'd) folders between the PC and CyberDuck file window in order to peform file transfers between your PC and cluster.

Command line scp/sftp

On Mac using the terminal program or on Windows using the PuTTy terminal, files can be transferred with the commands sftp or scp for example. The server name is eristwofs.partners.org and the default Port 22. Some examples of how to use these commands can be found here.

Command line rsync

Files and folders can also be synchronized between the cluster and the user's PC using the command line tool rsync. On the Mac this can be run in the terminal program or on Windows using the PuTTy terminal. As before, use eristwofs.partners.org as the server name with the default Port 22.

Command-line examples can be found here. For convenience, there are also gui client tools for this application e.g. rsyncosx

High-speed data transfer

Some transfer methods make better use of the available network bandwidth than others and are therefore faster for transferring large amounts of data. In this respect rsync is to be preferred over most methods, and users can run rsync in an eristwo/slurm batch or interactive job using the filemove partition (whose nodes are included in eristwofs.partners.org) to perform these large file transfers. More details about using the slurm job scheduler on eristwo can be found here.

For very large (>1TB) file sizes rsync can be run in parallel over ssh using a suitable number of cores e.g. parallel rsync

Screenshot of settings for CyberDuck

cyberDuckSettings.jpg

Go to KB0041651 in the IS Service Desk