There's no single answer.
Some algorithms are faster when dealing with small integers, because you can use small integers as the index into an array.
Some algorithms are faster when dealing with positive integers, because the set of positive integers has the property that it does not contain any infinite decreasing chains: any set of positive integers has a minimum.
Some algorithms are faster when dealing with integers because they assume a computational model where you can add or multiply integers in $O(1)$ time. It is dubious whether these algorithms are truly faster in practice, as that assumption isn't literally true, but they might be.
Some algorithms are faster (or simpler) when dealing with integers because integers can be not only compared, but also hashed, and the ability to hash items increases the range of options available to the algorithm designer. See the nuts-and-bolts problemthe nuts-and-bolts problem for an example vaguely of this form (there are probably better examples).
There are probably many other principles I'm not thinking of right now.
Most of the cases I know of involve only a $O(\lg n)$ speedup. However, be warned that asymptotic $O(\lg n)$ speedups are not very robust, for several reasons. Just because algorithm A is asymptotically faster than algorithm B by a factor of $\Theta(\lg n)$ doesn't mean it'll be faster in practice; it might be slower in practice, because of constant factors or because of considerations ignored in the theoretical analysis (e.g., caching effects, the memory hierarchy, and so on). Also, often the $O(\lg n)$ speedup results you see in this area will be dependent on a particular computational model and might not carry over to other computational models.
So, it's not clear how much weight you should put on this, if you are motivated by practical concerns.
See also:
Is there a meaningful difference between O(1) and O(log n)?Is there a meaningful difference between O(1) and O(log n)?
Logarithmic vs double logarithmic time complexityLogarithmic vs double logarithmic time complexity
Why is quicksort better than other sorting algorithms in practice?Why is quicksort better than other sorting algorithms in practice?
Explaining the relevance of asymptotic complexity of algorithms to practice of designing algorithmsExplaining the relevance of asymptotic complexity of algorithms to practice of designing algorithms
How can a quadratic algorithm be faster than a linearithmic one?How can a quadratic algorithm be faster than a linearithmic one?
Shouldn't complexity theory consider the time taken for different operations?Shouldn't complexity theory consider the time taken for different operations?
How can we assume that basic operations on numbers take constant time?How can we assume that basic operations on numbers take constant time?