Let's say I have a problem (i.e. Given f(x), find x) and two neural networks(i.e. feedforward and recurrent). I would like to know if one works better than the other one. I could run the twos on a computer, but other programs might interfere and I wouldn't know if the implementations I'm running are really the best ones humankind could create. Moreover, how could I be sure that the feedforward network worked better than the recurrent, when it might have just been "lucky"?

So, here is the question: can I compare the efficiency of two neural networks(with known sizes, structures and functions) from a theorical point of view? And if the answer is yes, how?

Thank you in advance.

  • 3
    $\begingroup$ Define "better". $\endgroup$ – Raphael Oct 11 '16 at 22:18
  • 1
    $\begingroup$ @Raphael or define f(x) ! $\endgroup$ – reuns Oct 13 '16 at 3:05

Essentially, no. The only way to know which neural network is going to give you better accuracy is to try them on a realistic data set. The theory we have is not well-enough developed to allow us to reliably predict which will do better on a particular data set.

A secondary remark. When you remark "other programs might interfere", that's not correct. Even if other programs are running (on a multi-tasking machine), they won't affect the accuracy. They might take the process of running or training the neural network on your data set take longer, but they won't affect the results.

  • $\begingroup$ That's exactly the problem: if I want to know which ANN reaches an average error of 10 percent in the shortest time, I have to consider the "noise" of other programs. Anyway, are there any other methods to compare two implementations with defined data sets, other than measuring the time to reach x accurancy? $\endgroup$ – samuelemarro Nov 23 '15 at 5:29
  • $\begingroup$ @samuelemarro, I appreciate your attempt to rephrase the question to make sure we're on the same page, but I'm afraid that this doesn't change my answer. The answer remains "no", in practice. $\endgroup$ – D.W. Nov 24 '15 at 0:47

I don't know if it is relevant to answer question after 1 year of being open, but here is my thought on the topic.

In order to avoid the "being lucky", I would advise to run the network on different parts of the data set.

I don't know if you already to this, but in order to improve your network, you start to divide your data in the training data (lets say 80% of the data) and the rest is your validation data.

The network will train itself on the training data, but since it is possible that overfitting occurs, you will have to check your network on the validation data and when this error is the lowest, you have with high probability the best network. Now, since this 80% of the data is random, do this multiple times for different sets and measure the calculation time for each of them. This might give you an averaged performance and thus a correcter conclusion.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.