I have on a few occasions trained neural networks (back propagation networks) with some rather complicated data sets (backgammon positions and OCR). When doing this, it seems that a lot of the work involves trying out different configurations of the networks, in order to find the optimal configuration for learning. Often there is a compromise between small nets that are faster to use/learn, and bigger nets, that are able to represent more knowledge.
Then I wonder if it could be possible to make some networks that are both fast and big. I'm thinking that at network where every neuron ain't fully connected ought to be faster to calculate than nets with full connection on all layers. It could be the training that detected that certain inputs are not needed by certain neurons, and therefore remove those connections. In the same way the training could also involve adding new neurons if some neurons seems to be "overloaded".
Is this something that have been tried out with any success ? Does any classes of networks exists with this kind of behavior ?