I have noted that the above expression can be rewritten as:
$$ A'_{i,j} = \alpha^i A'_{i,0} + \sum_{1 \leq j \leq N} (1 - \alpha) \alpha^{i-j}A_{i,j} $$$$ A'_{i,j} = \alpha^j A'_{i,0} + \sum_{1 \leq k \leq N} (1 - \alpha) \alpha^{j-k}A_{i,k} $$
Which makes me think I can define a $M \times N$ matrix $P$ as:
$$ P_{i,j} = \alpha^j A'_{i,0} $$
And a $N \times N$ matrix $R$ as:
$$ R_{i,j} = \begin{cases} \alpha^{j-i} & \text{if}~i \leq j \\ 0 & \text{otherwise} \end{cases} $$
And so I think that I would have:
$$ A' = P + (1 - \alpha) A R $$
Which would be a more parallelizable computation. The only challenges would be to efficiently construct $P$ and $R$ (which is probably not that hard) and, more importantly, the memory usage if $N$ is big (which unfortunately in my case it kinda is).