Can I say that cache miss penalty includes latency to memory? My current understanding is that cache miss penalty is the time moving data from the layer closer to main memory to it. But I'm not sure about what's the meaning of latency to memory? If they're not the same what's the overlapping part?
"Cache miss penalty" would be the extra time that is spent because of the cache miss.
Let's say your L1 cache has 5 cycles access time, and L2 cache has 23 cycles access time. That doesn't tell us enough yet.
A clever implementation might immediately detect that there is a cache miss, and send the data from the L2 cache simultaneously to the L1 cache and to whoever wanted the data. So without cache miss it might take 5 cycles, with cache miss 23 cycles. So you have 23 cycles latency for the L2 cache, but only 18 cycles miss penalty.
A less clever implementation might try the L1 cache first, then 5 cycles later start accessing L2 cache, put the data into L1 cache 23 cycles later, start the L1 cache access all over, and deliver the data another 5 cycles later. Total time 33 cycles, minus the 5 cycles without cache miss, so a 28 cycle miss penalty.