On-chip multithreading is considered a powerful performance enhancer technique that enables a processor to handle multiple threads at the same time. If I got the idea right, multithreading avoids pipeline stalls by processing instructions coming from different threads - since no data dependencies exist between different threads (in the textbooks, synchronization points are dropped for convenience), pipelines will never stall. However, authors like Tannenbaum state things like that (Structured Computer Organization, 6th Edition, page 562):
"... when a memory reference misses the level 1 and level 2 caches, there is a long wait until the requested word (and its associated cache line) are loaded into the cache, so the pipeline stalls. One approach to dealing with this situation, called on-chip multithreading, allows the CPU to manage multiple threads of control at the same time in an attempt to mask these stalls. In short, if thread 1 is blocked, the CPU still has a chance of running thread 2 in order to keep the hardware fully occupied."
I do understand that a pipeline will never stall if the data dependencies of subsequent instructions are eliminated (that is what out-of-order execution does), but if a memory reference is made by an instruction, the pipeline has to stall until the necessary information is retrieved for the main memory. I mean, no further instructions can be processed until the memory is read and the associated word is loaded. However, Tannenbaum and other authors suggest the opposite: while the word in question is being retrieved, additional instructions from other threads can be issued and that makes no sense to me. Can you folks clarify that? Shouldn't memory accesses prevent the pipeline from accepting new instructions, regardless the thread they come from? Thank you!