Wikipedia says that shared memory comes with lots of costs associated with cache coherence costs. But I thought the whole idea of shared memory is that all the CPUs access the same memory? So if one CPU changes that memory then other CPUs would access the same value? It would seem like this would require FEWER cache coherence costs? Is the idea that if one CPU changes its local cache before it writes to shared memory then other CPUs have to be notified?
If two different processors share one memory, each having individual cache, they can end up having two different values in the same address.
Imagine each of two processors has private caches L1 and L2. The cache L3 is shared between both processors. Assume the processor A reads data from address X in L3 to L1 and the processor B reads the same data from the same address (address X in L3) to it's private cache L1. Then, if the processor A modifies the value and does a write-back, the processor B can't figure it out without the support of coherence protocol and would still have an old value in it's own cache.
Basically, you are right. The cache coherence protocol is a mechanism to notify processors about shared memory modification caused by other processors.
The main advantage of the shared memory architecture for a programmer is that there is no need to explicitly describe communication and interaction between processors (like you would do using MPI, for instance). The coherence and consistency of the memory is fully the responsibility of the hardware.
You are correct: when one processor changes a memory location that is locally cached, then all other processors that are sharing that memory location need to be notified.
So why have local caches? If you didn't then every memory access would be as slow. You want most memory accesses to be fast, so each processor should have its own fast cache. This works extremely well for data that is used by only a single thread of computation (for example: each thread has a private stack, and no other thread will read or write that data.) Likewise it works extremely well for read-mostly data. For example, data structures that get initialized once and then read by many threads. The code of the program is also read only, so does well being cached.
For the few memory locations that really are shared, you need to keep the caches coherent. This involves detecting that a memory location is shared, and then notifying any of the other caches that are currently caching that memory location. You have to pay the cost of the book-keeping to detect which memory locations are shared even if you never actually use the shared memory.