L1 cache: Every time you access any data, either it is in L1 cache already, or one cache line worth of data (often 32 or 64 byte) is read from the next level, and stored into L1 cache. If necessary, the data that was stored in that cache line is removed, and if it was modified, it is first written to the next level of data.
Same with L2 cache; if data is not in L1 cache, then either it is already in the L2 cache, or it is read from the next level in exactly the same way. Same with L3 cache. On processors where CPUs and GPUs access the same memory, there is often another cache level between CPU / GPU and memory. So what exactly is in the various caches changes all the time.
Sometimes the programmer knows that putting data into cache is not worth it. For example, if I need to process one GB of data sequentially, which is much more than any cache, it is pointless to store the data into any higher level caches, because it will never be reused. For this reason, some processors have instructions to read data but not putting it into L2 or L3 cache.
And some processors have "streaming" instructions: If you know that you are accessing sequential data, once you read data at location x, the processor guesses that you will read data at the following cache line after this, so it starts reading data from memory and putting it into the cache before it is used. Sometimes it isn't used at all.