November 19, 2024
Users can map their ERISTwo home directories and briefcases to their PC desktops by accessing the file-sharing server eristwofs.partners.org. By doing so users can conveniently transfer files between their PC and the ERISTwo cluster. For Windows and Mac this process uses the smb protocol while for linux it is cifs. Examples of implementation will be given below.
We note that files can also be transferred to/from eristwofs.partners.org using SFTP (secure file transfer over SSH) and where an example of how to do this using an sftp client tool such as CyberDuck is illustrated below. This approach is to be preferred for large (>100GB) file sizes.
Data transfer speed considerations
Data transfer speed will be slow when using a wireless network, or the VPN - use a wired ethernet connection within the Mass General Brigham network if possible. Many wired ethernet connections at Mass General Brigham are limited to 100Mb/s. if you have a 100Mb/s connection and will be frequently transferring files to ERISTwo, contact helpdesk to request an upgrade of your connection to 1000Mb/s.
Transferring many small files is much slower than transferring the same data in a compressed archive such as a "Zip" file.
External hard drives are slower than the speed of the network and therefore limit the speed of data transfer. The maximum speed for USB (1) is 11Mb/s while USB2 is 480Mb/s.
The type of data storage drive in your computer makes a difference - Solid State drives (SSD) can read and write faster than 1Gb/s network transfer rates, but many Hard Disk Drives (HDD) limit data transfer to 320Mb/s or 640Mb/s depending on the age and size of the drive. Simple applications are available to test the read and write speed of your storage drive.
Mapping a network drive
All home and briefcase (group storage) directories are available to be mapped as network drives. If they so wish, the PI or data owner for a particular briefcase can submit a request to @email to restrict the mapped drive to read-only access.
From Microsoft Windows
Type the network path into the address bar of the File Explorer
- The network path to the home directory for a Mass General Brigham user abc123 would be
\\eristwofs.partners.org\homes\abc123
or possibly
\\eristwofs.partners.org\abc123
- The network path to data folder "/data/labName" would be
\\eristwofs.partners.org\labName
When connecting from a computer that is not a Mass General Brigham workstation, connect with your username, for example "PARTNERS\abc123"
From Mac OS X
Open the "Finder" application, select "Go" from the application menu, and "Connect to Server". Enter the server address:
- The server address path to the home directory for Mass General Brigham user abc123 would be
smb://eristwofs.partners.org/abc123
- The server address for data folder "/data/labName" would be
smb://eristwofs.partners.org/labName
Note: On some Mac OS X versions, using an address beginning with "smb://" does not work with our server, use "cifs://" instead.
From Linux
From the "Places" menu, select "Connect to Server" and enter the following details (using Mass General Brigham user ID abc123 as an example):
- Service type: windows share
- Server: eristwofs.partners.org
- Share: abc123 (or the name of a lab folder under /data)
- User: abc123
- Domain Name: PARTNERS
Using SSH/SFTP
GUI client tool
A convenient Graphical User Interface or GUI tool is Cyberduck which is available for both Mac and Windows platforms. For Windows users there is also the popular utliity WinSCP. On Linux platforms the OS will often provide a GUI sftp client tool as is found in "Gnome" desktop for example.
For configurring CyberDuck, enter the server name eristwofs.partners.org and your account details as shown in the screenshot below and press "Quickconnect".
When connecting for the first time you will be asked to accept the SSH key. Subsequently, drag and drop files or compressed (i.e. zipped or tar'd) folders between the PC and CyberDuck file window in order to peform file transfers between your PC and cluster.
Command line scp/sftp
On Mac using the terminal program or on Windows using the PuTTy terminal, files can be transferred with the commands sftp or scp for example. The server name is eristwofs.partners.org and the default Port 22. Some examples of how to use these commands can be found here.
Command line rsync
Files and folders can also be synchronized between the cluster and the user's PC using the command line tool rsync. On the Mac this can be run in the terminal program or on Windows using the PuTTy terminal. As before, use eristwofs.partners.org as the server name with the default Port 22.
Command-line examples can be found here. For convenience, there are also gui client tools for this application e.g. rsyncosx
High-speed data transfer
Some transfer methods make better use of the available network bandwidth than others and are therefore faster for transferring large amounts of data. In this respect rsync is to be preferred over most methods, and users can run rsync in an eristwo/slurm batch or interactive job using the filemove partition (whose nodes are included in eristwofs.partners.org) to perform these large file transfers. More details about using the slurm job scheduler on eristwo can be found here.
For very large (>1TB) file sizes rsync can be run in parallel over ssh using a suitable number of cores e.g. parallel rsync