November 5, 2024
General recommendations
- Use a wired network connection instead of WiFi whenever transfer speed is important
What is reasonable?
Typical speeds for different large-file transfers are:
| Source | Destination | Speed (Mbits/s) | Speed (MB/s) |
Time to transfer 1GB file |
Network |
| Local (scp) | SciC Linux Clusters | 707.2 | 88.4 | 12s | wired |
| Local (scp) | SciC Linux Clusters | 46.4 | 5.8 | 3min | wifi |
|
Local (Network drive) |
SciC Linux Clusters | 190.4 | 23.8 | 44s | wired |
| SciC Linux Clusters |
Local (Network drive) |
188.8 | 23.6 | 44s | wired |
|
Local (Network drive) |
HPCWin3 | 395.2 | 49.3 | 21s | wired |
| HPCWin3 |
Local (Network drive) |
426.4 | 53.3 | 19s | wired |
|
HPCWin3 (Network drive) |
SciC Linux Clusters | 310.0 | 38.8 | 65s | wired |
|
SciC Linux Clusters |
HPCWin3 (Network drive) |
305.0 | 38.1 | 70s | wired |
Factors affecting transfer speed
- Speed of internal hard drive
- File caching (repeat test several times)
- Speed of network connection
- Time of day / network congestion
- Size of files (transfer of many small files is slow)
- Transfer protocol
- SCP
- Windows network file share
- Rsync
Gather information about data speeds
For transfers over The Internet
Determine the network bandwidth to and from the Internet using SpeedTest.net or a similar public service. This should be performed both at the sending and receiving site.
- From a web browser: visit http://www.speedtest.net/
- From the command line:
wget https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
python speedtest_cli.py
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Partners HealthCare System (170.223.178.77)...
Selecting best server based on latency...
Hosted by Norwood Light Broadband (Norwood, MA) [18.87 km]: 8.785 ms
Testing download speed........................................
Download: 94.80 Mbits/s
Testing upload speed..................................................
Upload: 27.57 Mbits/s
Record the Download and Upload speed in Mbits/s.
Measure speed of data read/write to internal drive
On computers with hard disk drives (not solid state drives) the speed of data transfer to the drive can be the limiting factor in data transfer to the Scientific Computing resources. Free tools are available to measure the speed of data read/write to your hard drive
- Mac: Disk Speed Test
- Linux: Command line tools, including iozone. Graphical tool on Ubuntu
- Windows: hdtune.com