What roadblocks are there to HSA becoming standard, similar to floating point units becoming standard?

Question

I remember when my dad explained to me for the first time how a certain model of computer he had came with a "math coprocessor" which made certain math operations much faster than if they were done on the main CPU without it. That feels a lot like the situation we are in with GPUs today.

If I understand correctly, when Intel introduced the x87 architecture they added instructions to x86 that would shunt the floating point operation to the x87 coprocessor if present, or run some software version of the floating operation if it wasn't. Why isn't GPU compute programming like that? As I understand it, GPU compute is explicit, you have to program for it or for the CPU. You decide as a programmer, it isn't up to the compiler and runtime like Float used to be.

Now that most consumers processors (Ryzen aside) across the board (including smartphone Arm chips and even consoles) are SoCs that include CPUs and GPUs on the same die with shared main memory, what is holding back the industry from adopting some standard form of addressing the GPU compute units built in to their SoCs, much like floating point operation support is now standard in every modern language/compiler?

In short, why can't I write something like the code below and expect a standard compiler to decide if it should compile it linearly for a CPU, with SIMD operations like AVX or NEON, or on the GPU if it is available? (Please forgive the terrible example, I'm no expert on what sort of code would normally go on a GPU matter, hence the question. Feel free to edit the example to be more obvious if you have an idea for better syntax.)

for (int i = 0; i < size; i += PLATFORM_WIDTH)
{
    // + and = are aware of PLATFORM_WIDTH and adds operand2 to PLATFORM_WIDTH
    // number of elements of operand_arr starting at index i.
    // PLATFORM_WIDTH is a number determined by the compiler or maybe
    // at runtime after determining where the code will run.
    result_arr[a] = operand_arr[i] + operand2;
}

I am aware of several ways to program for a GPU, including CUDA and OpenCL, that are aimed at working with dedicated GPUs that use memory separate from the CPU's memory. I'm not talking about that. I can imagine a few challenges with doing what I'm describing there due to the disconnected nature of that sort of GPU that require explicit programming. I'm referring solely to the SoCs with an integrated GPU like I described above.

I also understand that GPU compute is very different than your standard CPU compute (being massively parallel), but floating point calculations are also very different from integer calculations and they were integrated in to the CPU (and GPU...). It just feels natural for certain operations to be pushed to the GPU where possible, like Floats were pushed to the 'Math coprocessor' of yore.

So why hasn't it happened? Lack of standardization? Lack of wide industry interest? Or are SoCs with both CPUs and GPUs still too new and is it just a matter of time? (I am aware of the HSA foundation and their efforts. Are they just too new and haven't caught on yet?)

(To be fair, even SIMD doesn't seem to have reached the level of standard support in languages that Float has, so maybe a better question may be why SIMD in general hasn't reached that level of support yet, GPUs included.)

Answer 1

A couple issues come to mind:

Synchronization/Communication overhead

In order to seamlessly transition from CPU to GPU code you need to communicate with the GPU. The GPU additionally has to be available(aka not rendering the screen), and all instructions on the CPU side of things need to retire/finish executing. Additionally you need to make sure that any pending writes have reached L3 cache/main memory, so that the GPU sees writes. As a result a transition to GPU code is quiet expensive, especially if the GPU is doing something latency sensitive(like rendering the next frame of something), and you need to wait for that process/task/thread/whatever to finish. Similarly returning back to the CPU is also expensive.

In addition you have to handle what happens if multiple CPU cores start fighting over the GPU.

Differing Memory Performance Needs

GPUs typically require high bandwith memory, but low latency is not as important, while CPUs are typically more sensitive to low latency. Low performance GPUs can and do use main memory, but if you wanted a high performance GPU built into the CPU you would potentially need two different types of memory. At which point there isn't much advantage to having everything on one chip, since all that does is make cooling harder.

Inertia/Dev Infrastructure

SIMD has compiler support right now and lots of work put into it. Simple GPU style workloads like dot products are already memory bound anyway on a CPU, so existing CPU+GPU combos would not benefit.

Could just have lots of SIMD

Not much more to say beyond heading. SIMD+Many cores+lots of execution units would give you a more GPU like CPU. Add better SMT for a bonus. See Xeon Phi for a real world implementation of this concept. Though one thing worth mentioning is silicon spent on more GPU style features is silicon not spent on branch prediction etc.

Edit:

Another thing that comes to mind is there are broadly speaking three reasons to have a GPU.

1. Just want to browse the web, display Netflix etc. For this use case existing CPU and GPU performance/architecture is more than sufficient.
2. Want to play high end videogames etc. Existing architecture has a lot of momentum behind it, and I'm not convinced gaming CPU workloads really need better SIMD performance, and instead need better cache/branch etc, though I don't really know. However the GPU is likely already busy so it might not be the best idea to shift even more work to the CPU
3. HPC applications. Custom hardware like Xeon Phi is available for people who need a more GPU like CPU.

Answer 2

Floating-point units are not standardised. Your typical Intel processor has at least two very very different ones built in. The results of floating-point operations are mostly standardised, but not completely.

But designing a floating-point unit is absolutely trivial compared to designing a GPU. GPUs are similar in complexity to CPUs, and CPUs are in no way standardized.