GPU works well or not mainly depends on computing instruction/IO instruction ratio. Here "IO instruction" includes any instruction that send/receive data through the boundary of the basic computation unit in GPU. "Basic computation unit" commonly have like 8-32 ALUs that need to execute instruction together and 16-128KB of registers/RAM and some instruction cache/RAM.
If the inner core of your program mainly fits in the instruction cache in the basic computation unit and the temporary data your proram need to read/write can mainly fit in to the registers/RAM in the basic computation unit, and the data to be feed in/processed and result to be exported is small enough , then you can utilize most of the computation power of GPU.
If not, then the grid/loop network connects the basic computation units and the GDDR memory interface and the PCIe CPU interface will soon render the GPU a CPU or even worse.
For example, each basic computin unit have 16 ALU, each need 2 register/RAM reader and 1 write with 32bit data, then each cycle need 3*16*2GHz*4bytes=384GB/s. The GDDR bandwith is only 300GB/s - not even enough to feed a single basic computin unit, and the GPU may have 100+ such basic computin units. All the magic from GPU is based on this, you need the 400GB/s*100+ bandwidth to make GPU work like magic. Once you need more "IO" bandwith, there is just not enough bandwith and the GPU magic vanish.
As @Bulat said, indeed this is the magic of any **U based on, you need to fit your temporary data mainly in L0 cache and don't let the IO bandwidth be the bottleneck to get nearer to the peak performance. L0 cache means multi-ported register or RAM that support through output of 1 instruction/circle which often need 2 read and 1 write.
Common programs for CPU which is commonly logic code instead of computing kernels mainly doesn't work at L0 cache speed, but at L2 or L3 or bus/DDR speed, this is the common working mode for CPU program and you don't think it is a bad thing for your program not to work at L0 speed on CPU.
But for GPU computing kernel working at L0 speed is the target. If the program doesn't work mainly at L0 speed, the program is less fit for GPU. But even so some program that doesn't fit can still work better on GPU than CPU, the final factor is how IO bandwidth is limiting the program.
CPU's L0 cache is the 8-16 32-64bit registers which is only 128 bytes. Although modern CPU have renaming register like 100+*64 bit, it is still jusg ~1KB and it is only utilized on specific instruction sequences. This is why for most time CPU doesn't work at the L0 speed, the L0 cache is very small only very special computing kernel can keep working at L0 speed. For CPU most code is control logic, you can never let control logic code mostly work at L0 even with more L0 cache so that is just a waste. And more L0 cache mean more registers then longer instructions to encode more registers then equally less instruction cache which is important for logic code. Larger L0 cache also means slower L0 speed - perhaps from 5GHz to 2GHz.
In contrast GPU provide 32-128KB L0 cache in each basic computing unit - hoping the code can run at L0 speed as much as possible, this is possible for small computing kernels.
Another factor of the GPU magic is GPU use more die size for basic computing unit. For example, CPU have 16MB L3 cache, GPU use this for 64KB L0 cache for 256 basic computation units (should be less since L0 cache cosume more area due to more port and control logic overhead). And GPU have lesser control logic to boost single thread performance.
Conclusion: CPU- fit for control code that work with 10MB code/data at L2/L3 speed. GPU - fit for computing kernel that work with 100KB data at L0 speed.
Note: the 100KB GPU L0 cache is devided into several ALUs. For example, 16ALU and 128KB L0 cache, then 8KB for each ALU, that is what your program can use to fit in L0. This introduces another pattern GPU need - your program have to be executing the same task on N different set of data then it can utilize N ALUs of the GPU at the same time. N at least should be larger than the number of ALUs in one basic computing unit.