For current multicore CPUs with support for SMT (Simultaneous Multi Threading), coarse-grained parallelism is strictly required, independently of the parallel technology used (pthreads, OpenMP, MPI). You need to feed each thread or process with enough work, otherwise the cost of thread creation/management and synchronization (for Pthreads and OpenMP) or the cost of communication and synchronization (for MPI processes) will be much higher than the work done per thread/process.
For GPUs, since these are actually SIMD units, i.e., data parallel machines, you need to feed them with data according to a fine-grained data decomposition (again, this is independent from the actual technology used, such as CUDA or OpenCL). In this case, thread management is lower (because it is done in hardware, not in software), but in general synchronization must be avoided, as much as possible. If you can not restructure your code so as to avoid synchronization you are going to experience bad performances on GPUS. Both NVIDIA and AMD recommend to avoid synchronization and to use GPUs for very simple data parallel tasks.
Moreover, you need to copy the data to be processed from the host to the GPU and back once results are computed, and this will also incur a performance penalty, depending on the size of your data processed by the GPU device.