Vectorization vs Asynchronous parallelism

Question

I have taken a course "Programming for Performance" in my college and in the first week of the course, I have come across vectorization and Asynchronous Parallelism. But I am unable to figure out what is the relation and difference between the 2 of them. In the slides, the professor has provided something like this-->

What does it actually mean that parallelism is finer-grained and course-grained?

Also here is another instance where he has tried to point out the difference, here we are given 2 code instance and tried to explain the concept, but I am not able to get that.

What I got is that, in the first one both vectorization and asynchronous parallelism is not possible, since there is a data dependency, but in the second one vectorization can be done as there is no dependency, but asynchronous parallelism is not achievable since, say we have this 2 instances

A(1) = A(2) + B(2) | A(2) = A(3) + B(3)

So in this case the A(2) may get overwritten by the second instance even before the 1st instance of the code is executed, hence Asynchronous Parallelism is not possible.

Is my understanding of this right?

Answer 1

Vectorization and asynchronous parallelism are not exactly comparable. Its like comparing apple and orange. Have a look at Difference between parallel, concurrent and asynchronous.

Vectorization uses the support for different vector instruction in modern processors. For e.g. Intel support AVX512 and AVX256 instructions. Here 512 and 256 are number of bits. So if the operand are 64 bit double values, the hardware can process 8 or 4 set of operands, parallelly. Operation(+,-,* ...) are applied on vector inputs (8 or 4 set of operands) instead of (8 or 4 times) on scalar (single set of operands) inputs.

for(int i=0;i<1000;i++)
for(int j=0;j<8;j++)
c[i][j]=a[i][j]+b[i][j]

On a Intel processor this code fragment will be replaced by vector instructions by the optimizing compiler . So it is equivalent to something like:

for(int i=0;i<1000;i++) { c[i][0]=a[i][0]+b[i][0] , c[i][1]=a[i][1]+b[i][1], ..., c[i][7]=a[i][7]+b[i][7] }

This require that there should be no dependency among consecutive iteration instructions. All eight addition operation execute in parallel by the vector processing hardware of the processor on eight different operand sets. Granularity in vectorization is at the level of 'instructions'.

Asynchronous operations can happen on even single core systems. For e.g., I/O processing. Yes it is the case that, two or more set of operations happen in parallel, but it is not necessary that both are computation operation. It could a memory operation and computation Or a I/O operation and computation. Granularity in Asynchronous operation is 'set of instructions' like a program or a interrupt vector routine.