I have noted that the above expression can be rewritten as:

$$
A'_{i,j} = \alpha^i A'_{i,0} + \sum_{1 \leq j \leq N} (1 - \alpha) \alpha^{i-j}A_{i,j}
$$

Which makes me think I can define a $M \times N$ matrix $P$ as:

$$
P_{i,j} = \alpha^j A'_{i,0}
$$

And a $M \times N$ matrix $Q = (1 - \alpha) A$. Then I'd have a $N \times N$ matrix $R$ defined as:

$$
R_{i,j} =
\begin{cases}
\alpha^{j-i} & \text{if}~i \leq j \\
0 & \text{otherwise}
\end{cases}
$$

And so I _think_ that I would have:

$$
A' = (1 - \alpha) A R + P
$$

Which would be a more parallelizable computation. The only challenges would be to efficiently construct $P$ and $R$ (which is probably not that hard) and, more importantly, the memory usage if $N$ is big (which unfortunately in my case it kinda is).