There are some known methods for doing this, but the important thing to remember is, we can only ever get an upper or lower bounds, but we can't know if they're tight. In general, determining the complexity of an algorithm is undecidable, since if we could do that, we could solve the halting problem.
But there are some known techniques for doing this, that will work well in practice. For example, if there's a loop i from 0 to n-1
, and the loop doesn't alter n
, or have any break statements, we know the loop will run exactly n
times.
For clear cut cases like this, you can compute this in a "top-down" way: you recursively determine the complexity of the loop body, determine the number of times the loop body is run. Things like this break down as soon as you do something more complicated (i.e. the number of times the loop is run changes, or doesn't depend directly on an input parameter), and there's really no way to know how they deal with hard cases without seeing their system.
Many systems are based on the idea of a system of recurrence relation: the complexity of a group of functions which call each other is determined, where the complexity of the functions called are left as variables. Then, you try to "solve" the system, or at least find an upper bound for it. This is particularly popular in functional languages.
Any bounds obtained this way aren't tight. What that means is, such an algorithm might return $O(\infty)$ even if the algorithm halts, return $O(2^n)$ even if the algorithm is polynomial (for upper bounds). Likewise, it might return $\Omega(1)$ for a lower bound, even if the algorithm is slower. These technically aren't incorrect as upper or lower bounds, they're just not tight.
It's also possible that they are doing some things empirically. For example, you could sample the run-time of the algorithm on a large number of inputs of different lengths. Then, you could set up a least-squares regression system for $n, n^2 ... \log n, n \log n, 2^n$, etc, and use some model-selection procedure to eliminate factors. This is pretty unreliable, though: lots of algorithms look polynomial for small inputs, and then turn exponential for larger inputs, plus there's lots of noise from the randomness of the procedure.