6
$\begingroup$

When copying ~80Gb data from one USB 3.2 stick to another, i noticed that the GUI of the transfer speed had clear periodicity. Out of curiosity, what could be the reason for this? No other active programs were running that, to my awareness, should put a periodic load on the system.

enter image description here

$\endgroup$
2
  • $\begingroup$ If your data is a collection of many instances of a smaller group of mixed files (ie: is periodic) then this can be another source of periodicity in the data transfer rate. For smaller files the per-file I/O overhead is significant with respect to the required data transfer time so the effective transfer rate goes down as file size gets smaller. This can be a bit of a fingerprint of the distribution of file sizes in a transfer. $\endgroup$
    – J...
    Nov 13, 2021 at 1:22
  • 1
    $\begingroup$ Related: if this were being transferred over a network, rather than between USB devices, there is an interesting "sliding window" algorithm in TCP which can have effects like this. It's actually known to screw up UDP video streams on the same network, because its hard for the UDP stream to figure out how much bandwidth is available... the TCP stream will keep interrupting it $\endgroup$
    – Cort Ammon
    Nov 13, 2021 at 1:33

2 Answers 2

11
$\begingroup$

Here's a possibility: Every transfer copies one megabyte and takes 0.4 seconds. The display is updated every second. The counter for "data transferred" is updated when a transfer is complete.

So 50% of the time the display is updated after 2 transfers with 2.0 megabyte completed, and 50% of the time it is updated after 3 transfers with 3.0 megabyte completed. That explains the "periodic" behaviour quite nicely.

In your case, your USB stick might be quite slow, so my numbers are actually realistic. In case of a hard drive: Hard drives have caches. So your computer can write say 100MB per second to the hard drive, but actually only 80MB get written and 20MB fill up the cache. Until the cache is full, and then your hard drive doesn't accept new data until say 100MB of the cache are written to the actual hard drive. That will look to your computer as if write speed wasn't constant.

But since you posted on CS, it's now your duty to explain exactly what happened :-) For example, you could write a program that logs reading and writing with the exact times when they happen.

$\endgroup$
4
  • 2
    $\begingroup$ This was my first thought also. About 90% of the times that I’ve seen oscilating transfer throughput with a stable frequency like this, it’s been a case of harmonic interaction between the sampling rate and the transfer frequency. $\endgroup$ Nov 11, 2021 at 22:20
  • $\begingroup$ This is a reasonable suggestion. I will do some testing. This transfer was many small files with identical size. The USB sticks are brand new USB3.2, so writing speed was also waaay slower than expected. $\endgroup$
    – Paamand
    Nov 12, 2021 at 15:07
  • $\begingroup$ Many USB sticks are quite slow. And there are USB sticks tha claim to have 64 GB capacity but have one 2GB. You don’t notice when you write, it just overwrites earlier data, just when you read, the old data isn’t there.., $\endgroup$
    – gnasher729
    Nov 12, 2021 at 15:14
  • $\begingroup$ @Paamand Actually, if it’s mostly small files, that type of transfer speed is well within what I would expect. Bulk transfer of a bunch of small files almost always runs significantly slower than transferring the same amount of data as one big file due to the overhead associated with creating files. $\endgroup$ Nov 12, 2021 at 18:08
4
$\begingroup$

The transfer is DMA so it isn't really the other programs which create periodicity. It is more like bus-mastering and the speed with which the device answers the read and write commands. I assume you are on Windows desktop with an x86-64 CPU. On these desktops an xHCI is used to transfer data. I actually wrote a partial xHCI driver which I have not finished yet.

I can tell the xHCI supports MSI-X. The xHCI thus can trigger interrupts directly to the local APIC of one specific CPU core. The currently running program on the core is interrupted and the TSS is used to switch the stacks if there was a ring 3 to ring 0 transition. Otherwise, the interrupts are simply nested on the same stack. The interrupts are short lived because they don't do the main work. They simply do some work to add the interrupt to the queue of programs that the OS will then switch on using the scheduler. The data Transfer Event TRB is probably also read by the driver and it pushes the data of the TRB to some other place. This is called Bottom-Half Interrupt Processing. The TRB informs software of the size of the transfer (for more info see When are a controller's registers loaded and ready to inform an I/O Operation?).

The periodicity can be created by file switching. One file can be "programmed" to be read/written to a USB stick to a specific address at once whatever the file's size) but you cannot tell the xHCI to just write to a device with five different addresses and tell it to read the files from 5 different addresses. There is some processing involved by software on file switch that can create periodicity in the transfer.

The scatter gather operations can create periodicity because the virtual memory is contiguous but not physical memory. The physical memory is separated in chunks of 4KB so the buffer allocated to your program (the built-in file explorer) may be bigger than a page and be scathered around RAM. To avoid that you can use one buffer of 4KB but that makes the operation slower because you need to schedule the operation several times to read into the buffer. Instead, file explorer probably makes the buffer much bigger and let the OS do the scather gather by putting several TDs on the Transfer Ring of the endpoint for that device. The TDs contain TRBs and specify physical addresses so the OS must make sure that the transfer doesn't overwrite parts of the address space of another process. It must also pin the pages to physical memory instead of swap because DMA transfers are writing directly to it. If the OS swaps the page-frame to the hard-disk and give the page-frame to another process, the DMA engine is writing to another process's address space. This is very dangerous and could be a potential vulnerability. There have been some vulnerabilites known in the past so this is actually something that can happen in a recent OS (because it happened in recent years but I don't know if there are current vulnerabilities).

A lot of other things can create periodicity. It is a very complex operation which involves bringing the data to RAM and then sending the data back to the other device.

Also, I didn't read the xHCI spec for the bulk transfer part. Bulk transfers seem to be the way to transfer data from a mass storage device to RAM or vice-versa. Maybe start to read the spec and you'll have all the info you need. The spec is linked on my answer that I linked above.

$\endgroup$
5
  • $\begingroup$ Surely this would happen with a periodicity of milliseconds or microseconds $\endgroup$
    – user253751
    Nov 12, 2021 at 12:15
  • $\begingroup$ Thanks for this nice technical answer. However I lean towards the other answer given since the periodicity here is on a larger timescale (seconds) that I cannot see fit with your suggestions (which may also happen). $\endgroup$
    – Paamand
    Nov 12, 2021 at 15:04
  • $\begingroup$ In general, file switching is a major burden on file transfers. It involves several disk IO for each components of the file's path which all have to be scheduled and run eventually. Down the line, one transfer requires the last so the OS must wait for some transfers to be done. $\endgroup$
    – user123
    Nov 13, 2021 at 22:00
  • $\begingroup$ This is why if you have a lot of small files the transfer will transfer lots of items/s but have a very low bytes/s transfer speed $\endgroup$
    – user123
    Nov 13, 2021 at 22:04
  • $\begingroup$ It isn't really "periodic" like in your question though. $\endgroup$
    – user123
    Nov 13, 2021 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.