I just finished reading a white-paper from a recent AI startup. The company, Deep Brain Chain, wants to distribute neural network training over computers worldwide, using blockchain technology. Here is the whitepaper
From my understanding of machine learning and artificial neural networks, the communication bandwidth is the biggest bottleneck. I would assume that anyone needing infrastructure AI services is using some pretty big models. The bigger the training batch, the more likely it is that data transfer speed is a bottleneck.
Now, attempting to paralyze this across multiple nodes "in the cloud" does not seem like it would scale well. Far, far from linear at least.
Utilizing a distributed network of blockchain mining computers seems like an additional step towards imperfect parallelization. The internet connection speed is bound to be 10-100x times slower than speeds achievable in a cloud datacenter.
Plus, none of the computers in a distributed network of blockchain miners are running the same hardware specifications. I'm not sure, but this seems to add lots of difficulty to the parallelization and efficiency issues.
It is my understanding that neural networks are not highly parallelizable to begin with, much less suited for this type of distribution.
However, I've come here to ask these questions: Are there any (known) training algorithms that can avoid the problems I've posted above? Can there be distributed neural network training systems that utilize a massive network of home PCs and data centers alike?