The intermediate language is a relatively normal programming language and as such you can write an optimizer for it. Optimization passes like common subexpression elimination, constant propagation, or strength reductions are fairly independent of the programming language.
As long as the intermediate language can express all concepts from the source language reasonably well, there is little loss in just optimizing the intermediate representation. For example LLVM's intermediate language is well suited for representing C-like imperative languages. It also comes with a type system, so not much information is lost when translating from C, or Ada, or Java to LLVM.
There is of course a problem if the source language contains information that can't be encoded in the intermediate language. For example you could think about just optimizing x86 assembly instead of first translating to something like LLVMs IR. However assembly lacks many high level constructs that make it easier for the compiler to optimize things. For example it lacks a type system. If your Ada code restricted the value of a variable to be less than five, the compiler could use that information to optimize some conditionals. Such information would be lost when translating to Assembly.
Because of this mismatch it is not uncommon to first do some high level optimizations in the target language and then translate to the intermediate language to leverage all the work that went into the optimization passes that are suitable for many languages. This article about Rust's Mid-Level IR provides an example of this. But this step is not strictly necessary (For example Rust didn't do this at first), you just lose some optimization opportunities if you skip it.
In the end it's probably an engineering tradeoff. You lose a few opportunities for optimization by using an intermediate language. But you save enormous amounts of developer time by reusing existing optimization passes.