I have read that a decrease in response time does not necessarily entail an increase in throughput, but I'm struggling to understand the exact way that this can happen. What I'm imagining is a server receiving requests from several different connected devices. Say computer
A sends a request and the server starts working on it. While it's working, it receives a request from computer
To keep the model simple, we imagine that while the computer is processing request
A, it holds request
B and will get to it when it's done sending the response to
A takes 10 seconds to compute 1 billion flops or whatever.
B takes 10 seconds to compute 2 billion flops because maybe those flops for some reason required more processing or something. And maybe
B was received 5 seconds after
So the net time that
A spent waiting is 10s, and the net time
B spent waiting is 15s, so the average response rate is I guess is
$$ 25s/3\ billion\ flops $$
(I don't know that this is the right way to think about response rate but it's the best I can come up with.)
And the throughput is, I guess proportionate to
$$ 3\ billion\ bits / 20s $$
(if the number of bits about linearly correlates to the number of flops, which I think is reasonable?)
Now if you double the response rate, the number of bits doesn't change but the server now spends just 5s on
A and receives
B's request right when it's done with
A, spending 5s on that. The response rate is now
$$10s/3\ billion\ f$$
The throughput is proportionate to
$$3\ billion\ b/10s$$
So the response rate more than halved, but the throughput doubled.
So what could happen to make the throughput stay fixed while the response rate doubles? Any examples that I can find while googling seem to vaguely indicate that the server could increase the response rate by distributing the time that it spends on each request evenly while not affected the throughput.
But I don't see how that increases the response rate -- isn't the response rate a weighted average over all requests? If not how could you even talk about "the response rate" when every particular request will make a different demand and require a different amount of time? If it is an average, then the order in which requests are processed shouldn't matter since the net time and net flops isn't affected.