If one attempted to download a file at a speed of 800 Mb/s (100 MB/s) onto a hard drive with a write speed of 500 Mb/s (62.5 MB/s), what would happen? Would the system cap the download speed?
$\begingroup$
$\endgroup$
5
-
5$\begingroup$ This is, and has been since the inception of networks, a totally common situation when you are in a fast network, like in a university outside of the busy hours. $\endgroup$– Peter - Reinstate MonicaCommented Nov 27, 2017 at 8:17
-
4$\begingroup$ Since you can program, you can easily observe this yourself. Write a simple TCP server and client (you should be able to find examples for the socket library of your language of choice), let the serve deliver data as fast as it can, and modify the client to read that data only at a specified speed (sleep between reads). Check that the data is received intact. You can even observe the packets sent with wireshark or similar. $\endgroup$– Carsten SCommented Nov 27, 2017 at 10:25
-
3$\begingroup$ Having had an experience as described in @PeterA.Schneider's post, I can tell you that you have to take into account the RAM buffers of the operating system. Until they are filled, you effectively can store away the downloaded data at the RAM's speed. (Usually in thousands of megabytes per second.) $\endgroup$– Nobody moving away from SECommented Nov 27, 2017 at 14:01
-
$\begingroup$ @Nobody And, of course, there's always the absurd pipes that make up the internet backbone which transmit data so fast that your typical computer's RAM can't even keep up! $\endgroup$– Cort AmmonCommented Nov 27, 2017 at 19:04
-
$\begingroup$ Your computer would genarate ICMP request telling it to slow down. However, at these meager speeds you might as well get a SSD or M.2 ssd and now your write speed are so high 100mb/s will be easy. Some M.2 are 1500MB/s write. Put 2 or more in a RAID and you can multiple that speed many time over. plain SSD can be 300-450MB/s depending on capacity and brand. storagereview.com/samsung_960_pro_m2_nvme_ssd_review $\endgroup$– cybernardCommented Nov 28, 2017 at 4:51
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
14
Many protocols, including TCP which is most widely used protocol on the Internet, use something called flow control. Flow control simply means that TCP will ensure that a sender is not overwhelming a receiver by sending packets faster than it can empty its buffer. The idea is that a node receiving data will send some kind of feedback to the node sending the data to let it know about its current condition. So, two way feedback allows both machine to optimally use their resources and prevent any problems due to mismatch in their hardware.
-
2$\begingroup$ ... ideally. Bufferbloat means that it is likely that a several switches will be storing and waiting to forward an ever-growing pile of packets (which might be mitigated somewhat by a transfer protocol that requires occasional ACKs to continue). $\endgroup$ Commented Nov 27, 2017 at 1:48
-
3$\begingroup$ You could mention that this distinguishes TCP (hey, it has control in its name) from UDP which will happily keep on sending, regardless of consumer capacity. $\endgroup$ Commented Nov 27, 2017 at 8:16
-
6$\begingroup$ @Fabian This is what happens all the time anyway. The OS's hard drive cache allows data to be 'sent' to the HDD much faster than it can be written, filling the in-RAM cache and writing to disk as soon as possible. So, depending on the amount of (unused) RAM in your machine, the OS may buffer some dozens to hundreds of MB of a file in RAM before it reaches the disk. Cache size is however a compromise, because data which the application may consider to be already persistent on disk may not be for a certain time which may cause unexpected data loss in case of (power) failure or similar. $\endgroup$– JimmyBCommented Nov 27, 2017 at 12:07
-
2$\begingroup$ @Fabian : this you can see in the file copy dialog in Windows, from 8.1 onward. In expanded mode, it shows the transfer rate, and you can observe, that the first half of a second or so is very fast, and if the file is larger than what could have been transferred during that, then the speed drops significantly, to a constant rate. That fast speed at the beginning was due to the cache. After it is filled, the speed is reduced. $\endgroup$– vszCommented Nov 27, 2017 at 12:23
-
1$\begingroup$ @EricTowers No, bufferbloat only happens when a network connection is the bottleneck. (You might see a form of bufferbloat at the hard drive, that will slow down other hard drive operations, but it won't affect other network connections) $\endgroup$ Commented Nov 28, 2017 at 0:35
$\begingroup$
$\endgroup$
what would happen?
- The bytes that cannot be written to the HDD in time would be buffered temporarily somewhere; very likely some of them would be buffered in the application that "converts" the network traffic into HDD traffic (i.e., your browser), and in cases of longer congestion, the lower level parts of the stack (i.e., the networking subsystem of your PC) would have buffers at lower levels as well, which would fill up until full.
- The information that the data is coming too fast would then propagate back to the sender in some fashion. In the case of TCP/IP, this would mean that there is a part of the TCP protocol which tells the sender that it should stop sending. The Wikipedia page on TCP/IP says it best when it says:
Its [TCP] responsibility includes end-to-end message transfer independent of the underlying network, along with error control, segmentation, flow control, congestion control, and...
- At the end of the day, the "system" (not necessarily the computer of the receipient, but the total system consisting of sender, network and receipient) would indirectly "cap" the download speed. At no point is a "speed" variable in existance, though. Both network traffic and HDD traffic is block-based, that is, the delays between those blocks would get long enough (on the faster network side) until the net "speed" (throughput) is on average the same as the slower speed of your HDD.
Would the system cap the download speed?
Not literally. A few dozen years ago, computers indeed communicated with synchronous speeds (i.e., modems with fixed speed settings). This is, at some relatively low level, also happening today; i.e., your DSL or cable modem will probably have some set speed at which it communicates with its direct physical uplink component. But in 2017, all of this is sufficiently complicated that almost all components are quite dynamic. In the good old days, modems were often physically only able to speak a certain speed, and that even had to be established before dialing in. There are (were) some network protocols like ATM which tried to get a synchronous wide-area network, but they didn't turn out so well, compared to TCP/IP.
Today, the effective speed is usually always implicitely reached by buffering, flow control, and such.
Also note that in your example there can even be more participiants. The network interface could have more work to do (other connections to your PC). Some part of the network along the way could be busy (a family member surfing or watching videos). The HDD might be doing 3 large file writes at the same time. So it would make no sense to specifically set a "speed" which could be capped.
$\begingroup$
$\endgroup$
File downloads (usually) happen over a protocol called TCP. In TCP, the sender doesn't send any data until the receiver is ready to receive it.
What will happen is that your computer will reserve some amount of RAM to hold the received data temporarily before writing it to your hard drive (this is called a buffer). Then it will ask the sender for just enough data to fill the buffer. As it receives data, it will start writing data from the buffer to the hard drive - however, it won't ask for more data while the buffer is nearly full.
The overall effect is the sender simply waits for the receiver to catch up before it continues sending.
$\begingroup$
$\endgroup$
Try to copy from SSD disk to HDD disk. What will happen? The connection will follow the slowest part. However, I have perceived network bottleneck. Get 10000 torrents to download "1kb/s". What happens? "Error on the disk".... Cause disk head on plate cannot go in so many different places per second. So, I think that HDDs have shown the age of their speed and people with fast internet speed may perceive occasional silent network bottlenecks due to that.
Note that internet network clients may cache files on memory. When Full (doing other things will make mem to get full quicker), both the net clients and other process of machine will be cached in Disk.If only one HDD disk is in place, and network traffic is going on this disk, and temporary cache on the same disk, then it will be further slowed down. A disastrous result.
Net cache on ram --> Ram full -> Cache on Disk -> same disk that net traffic goes also make temp cache of ram -> Further slowed down --> Disastrous result --> the whole system will be slowed down.
(It almost out of question to copy/move files on the same disk that receiving a lot of internet traffic).
SSds is a solution if safety protocols of data loss are in place. But they are sadly, very expensive yet.
So, i believe we live in an epoch that people with fast internet connection silently have Network speed data transfer loss due to plate-HDDs.
I have noticed my mobile phone can download much faster than my computer (SD cards are essentially SSD type).