Throughput and response time

Question

I have read that a decrease in response time does not necessarily entail an increase in throughput, but I'm struggling to understand the exact way that this can happen. What I'm imagining is a server receiving requests from several different connected devices. Say computer A sends a request and the server starts working on it. While it's working, it receives a request from computer B.

To keep the model simple, we imagine that while the computer is processing request A, it holds request B and will get to it when it's done sending the response to A. Task A takes 10 seconds to compute 1 billion flops or whatever. B takes 10 seconds to compute 2 billion flops because maybe those flops for some reason required more processing or something. And maybe B was received 5 seconds after A.

So the net time that A spent waiting is 10s, and the net time B spent waiting is 15s, so the average response rate is I guess is

$$ 25s/3\ billion\ flops $$

(I don't know that this is the right way to think about response rate but it's the best I can come up with.)

And the throughput is, I guess proportionate to

$$ 3\ billion\ bits / 20s $$

(if the number of bits about linearly correlates to the number of flops, which I think is reasonable?)

Now if you double the response rate, the number of bits doesn't change but the server now spends just 5s on A and receives B's request right when it's done with A, spending 5s on that. The response rate is now

$$10s/3\ billion\ f$$

The throughput is proportionate to

$$3\ billion\ b/10s$$

So the response rate more than halved, but the throughput doubled.

---

So what could happen to make the throughput stay fixed while the response rate doubles? Any examples that I can find while googling seem to vaguely indicate that the server could increase the response rate by distributing the time that it spends on each request evenly while not affected the throughput.

But I don't see how that increases the response rate -- isn't the response rate a weighted average over all requests? If not how could you even talk about "the response rate" when every particular request will make a different demand and require a different amount of time? If it is an average, then the order in which requests are processed shouldn't matter since the net time and net flops isn't affected.

Answer 1

You have to define what you mean by "response time".

Let's suppose we have a client sending a request to a server over a network, and the server sending back a response. What is the time it takes from when the client has the request to when it receives the response? Well, there likely is the time the request spends on a queue waiting for the network to become available, the time it takes for the request to transit the network, the time the request spends on a queue waiting for the server to become available, the time it takes for the server to compute a response, the time the response spends on a queue waiting for the network to become available, and the time it takes for the response to transit the network. Response time might mean the sum total of all of those, or it might mean just one of those quantities (e.g., the time it takes for the request or the response to transit the network, or the time it takes for the server to compute the response).

If the time it takes for the response to transit the network decreases, but the bottleneck is the server's ability to handle requests, then the overall throughput will not increase, because the server's throughput will be the bottleneck. You will get responses back a little bit quicker but the number of responses received per second will remain the same.