What can be learned from the weights in a neural network?

Question

I'm very new to neural networks, and have been trying to figure some things out. So, let's say you come across a neural network which has 100 inputs, a hidden layer with 200 nodes, and 32 outputs. Let's also say that you, the "discoverer" of this particular instance of a neural network, are able to read the weights of the individual neurons. What could you figure out about it's function?

1) Are you able to determine what the algorithm or logic is contained within the neural network? Other than feeding in all possible inputs and studying the outputs it produces.

2) If you were given information about the connection of the neural network (maybe the network isn't fully connected), would solving question one above be easier?

Answer 1

It depends. Weights of neural networks can be graphed or visualized for some insight. This is especially useful if the neural network works with visual processing. It is possible to "derive" what low-level inputs to the neural network create particular neurons in higher levels to "fire" by working backwards through the neural network weights— in other words, the problem of finding/deriving the low-level input patterns that maximally excite particular neurons, and graphing the results. A great example of this is the recent breakthrough result by Google in a self-trained visual network that self-organized to find higher-level patterns like cats and human faces, etc. [1],[2],[3]

This is also known as "feature detection" and there is Nobel-prize winning research (1981 Hubel/Weisel) that demonstrates that real brain neurons function in a similar way, to varying degrees. Active research is ongoing/continuing in this area in both biological and artificial systems.

Another way of analyzing neural net weights is to conclude what factors (inputs) affect the neural network and which don't. For example, suppose the neural net is used to predict stock prices and it has various inputs relating to different economic variables such as say GDP, gold prices, DJIA (an index), and interest rates. After the network is trained (successfully!) to predict something (say future prices), one can determine how much effect each one of the input variables has on the final prediction.

Also a determination can be made of basic negative or positive correlation between input and output. In this way neural networks can be used in a manner very similar to statistical techniques like factor analysis.

So the answer is "yes absolutely," but only in the sense that there are different ways to reveal "algorithms" in neural networks via graphical or other "human-readable" representations other than with the typical representation of algorithms, i.e. code. But representing neural network weights in a human-readable way and finding new useful representations is an active area of research.

[1] Google Puts Its Virtual Brain Technology to Work Technology Review

[2] Google’s Artificial Brain Learns to Find Cat Videos Wired

[3] How Many Computers to Identify a Cat? 16,000 NYT

Answer 2

Are you able to determine what the algorithm or logic is contained within the neural network? Other than feeding in all possible inputs and studying the outputs it produces.

No, I don't think so, not in a meaningful way. That would be akin to studying the bits in each individual byte of a computer program in an effort to evaluate its purpose. You need meaning to determine that, and you can only get that by studying the inputs and outputs, or evaluating actual opcodes.

There's no meaning in individual neuron weights; it's only when those weights are combined into an answer that they become meaningful.

You could probably ascertain the training method of the neural net by observing its overall structure and the pattern of relative weights in the neuronal structure.