HOWTO: Optimizing Data Transfer Speed over the Mass General Brigham Computer Network or the Internet

General recommendations

• Use a wired network connection instead of WiFi whenever transfer speed is important

What is reasonable?

Typical speeds for different large-file transfers are:

Factors affecting transfer speed

• Speed of internal hard drive
• File caching (repeat test several times)
• Speed of network connection
• Time of day / network congestion
• Size of files (transfer of many small files is slow)
• Transfer protocol
  
  • SCP
  • Windows network file share
  • Rsync

Gather information about data speeds

For transfers over The Internet

Determine the network bandwidth to and from the Internet using SpeedTest.net or a similar public service.  This should be performed both at the sending and receiving site. 

• From a web browser:  visit http://www.speedtest.net/

• From the command line: 

wget https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
python speedtest_cli.py 
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Partners HealthCare System (170.223.178.77)...
Selecting best server based on latency...
Hosted by Norwood Light Broadband (Norwood, MA) [18.87 km]: 8.785 ms
Testing download speed........................................
Download: 94.80 Mbits/s
Testing upload speed..................................................
Upload: 27.57 Mbits/s

Record the Download and Upload speed in Mbits/s. 

Measure speed of data read/write to internal drive

On computers with hard disk drives (not solid state drives) the speed of data transfer to the drive can be the limiting factor in data transfer to the Scientific Computing resources.  Free tools are available to measure the speed of data read/write to your hard drive

• Mac: Disk Speed Test
• Linux: Command line tools, including iozone. Graphical tool on Ubuntu
• Windows: hdtune.com