HOWTO: Optimizing Data Transfer Speed over Partners Network or The Internet

General recommendations

  • Use a wired network connection instead of WiFi whenever transfer speed is important

What is reasonable?

Typical speeds for different large-file transfers are:

Source Destination Speed (Mbits/s) Speed (MB/s)

Time to transfer

1GB file

Network
 Local (scp) ERISOne  707.2 88.4   12s wired
 Local (scp) ERISOne  46.4  5.8  3min wifi

 Local 

(Network drive)

 ERISOne  190.4  23.8  44s wired
ERISOne

Local

(Network drive)

188.8 23.6 44s wired

Local

(Network drive) 

 HPCWin3  395.2  49.3  21s wired
HPCWin3

Local

(Network drive)

426.4 53.3 19s wired

 HPCWin3

(Network drive)

 ERISOne  310.0  38.8  65s wired

ERISOne

 HPCWin3

(Network drive)

305.0  38.1   70s wired 

Factors affecting transfer speed

  • Speed of internal hard drive
  • File caching (repeat test several times)
  • Speed of network connection
  • Time of day / network congestion
  • Size of files (transfer of many small files is slow)
  • Transfer protocol
    • SCP
    • Windows network file share
    • Rsync

Gather information about data speeds

For transfers over The Internet

Determine the network bandwidth to and from the Internet using SpeedTest.net or a similar public service.  This should be performed both at the sending and receiving site. 

  • From the command line: 
wget https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
python speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Partners HealthCare System (170.223.178.77)...
Selecting best server based on latency...
Hosted by Norwood Light Broadband (Norwood, MA) [18.87 km]: 8.785 ms
Testing download speed........................................
Download: 94.80 Mbits/s
Testing upload speed..................................................
Upload: 27.57 Mbits/s

Record the Download and Upload speed in Mbits/s. 

Measure speed of data read/write to internal drive

On computers with hard disk drives (not solid state drives) the speed of data transfer to the drive can be the limiting factor in data transfer to the Scientific Computing resources.  Free tools are available to measure the speed of data read/write to your hard drive


Related articles