Is there a methodical procedure for determining the optimal timeout / retry strategy for dealing with a remote server that handles processes responses for requests, given some probability distribution of response time?

For example, suppose I have a client program which requests a web page from some server, and I know roughly the probability distribution of its response time, e.g. 95% of the time I'll get a response with a gamma distribution, and 5% of the time my request will get lost and a response never sent. So maybe 80% of the time I'll get a response within 100 milliseconds, and 90% of the time I'll get a response within 500 milliseconds.

On one hand, I could set a timeout of 500 milliseconds and retry with another request after then. Or I could set a timeout of 100 milliseconds. A longer timeout means I have more chance of success, but I have to wait longer. A shorter timeout means I can reduce my wait time potentially, at the cost of increasing the chance of failure (I give up too early; the server will respond). Or I could just send two identical requests up front and take whichever response arrives first -- this seems like it might be the optimal strategy, but something seems wrong here. (why then wouldn't I send 100 identical requests? I guess I'm not modeling the fact that lots of additional requests would clog the server queue.)

Please help; I know this is a fairly general question, but I have no idea where to look for more information of this type. I never took any classes on distributed systems in college + am trying to improve responsiveness of a client program.


The simplest solution I have found to this problem is using rolling percentile latency. The optimal timeout will shift based on hardware, network configuration and the state of the operating system. Therefore it cannot be fixed/established beforehand, but approximated at runtime. The typical practice in industry is to maintain P95 and P99 response times. These are durations in which 95% and 99% of requests complete respectively over some time duration. So you could perhaps store the last hour of request times, and if the current request is longer than 95% of the others in the last rolling 60 minutes, then you deem that a timeout.

This will adapt and gradually become more strict if response times are fast, and more lenient if they become slow.

You can read more about percentile latency here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.