28

Temperature is a hyperparameter of LSTMs (and neural networks generally) used to control the randomness of predictions by scaling the logits before applying softmax. For example, in TensorFlow’s Magenta implementation of LSTMs, temperature represents how much to divide the logits by before computing the softmax. When the temperature is 1, we compute the ...


21

No. This direction is unlikely to be useful, for two reasons: Most computer scientists believe that P $\ne$ NP. Assuming P $\ne$ NP, this means there does not exist any polynomial-time algorithm to solve any NP-complete problem. If you want your neural network to solve the problem in a reasonable amount of time, then it can't be too large, and thus the ...


17

Neural Networks are not magic. If you treat them like they are and just throw data at them without thinking you're going to have a very bad time. You need to stop and ask youself "Is milliseconds since 1970 actually going to be predictive of the event I'm interested in?" The answer you should arrive at immediately is no. Why? For every instance you ...


14

Artificial neural networks is a class of algorithms that include a lot of different kinds of algorithms based on graphs, so I won't detail here beyond what you asked because there's too much to say, since there are so many kinds of ANNs. The first kind of artificial neural networks, the famous McCulloch-Pitts neurons, were linear, meaning that they could ...


13

(I assume by "error propagation" you mean what I call "error back-propagation.") On page 231 of Neural Networks (by Haykin), he states that back propagation always converges, although the rate can be (in his words) "excruciatingly slow." I think what you are asking though is not whether the algorithm will always converge, but whether it will always ...


13

A perceptron is always feedforward, that is, all the arrows are going in the direction of the output. Neural networks in general might have loops, and if so, are often called recurrent networks. A recurrent network is much harder to train than a feedforward network. In addition, it is assumed that in a perceptron, all the arrows are going from layer $i$ ...


12

Just a note: rational-weighted recurrent $NN$s having boolean activation functions (simple thresholds) are equivalent to finite state automata (Minsky, "Computation: finite and infinite machines", 1967); rational-weighted recurrent $NN$s having linear sigmoid activation functions are equivalent to Turing Machines (Siegelmann and Sontag, "On the ...


12

Yes it can, and has been. In the paper Map-Reduce for Machine Learning on Multicore they discuss using the Map-Reduce paradigm for several common ML algorithms including ANNs.


11

Short answer: Strictly speaking, "Deep" and "Spiking" refer to two different aspects of a neural network: "Spiking" refers to the activation of individual neurons, while "Deep" refers to the overall network architecture. Thus in principle there is nothing contradictory about a spiking, deep neural network (in fact, the brain is arguably such a system). ...


10

This sounds more like the makings of a computational model rather than a programming language, as such, perhaps in the same way that the quantum computation can form the basis of a programming language such as the quantum lambda calculus. Ask yourself: What kinds of computation are you trying to perform? How can these computations be composed? How can the ...


10

artificial neural networks models were generally restricted to only a few layers, say 3, for decades, for various reasons, including a math proof named Kolmogorovs thm that indicated they could theoretically approximate arbitrary functions with low error (but only with many neurons). multilayer networks beyond that were not feasible/effective via prior ...


10

Using Backpropagation with momentum in a network with $n$ different weights $W_k$ the $i$-th correction for weight $W_k$ is given by $\Delta W_k(i) = -\alpha \frac{\partial E}{\partial W_k} + \mu \Delta W_k(i-1)$ where $\frac{\partial E}{\partial W_k} $ is the variation of the loss w.r.t. $W_k$. Introduction of the momentum rate allows the attenuation ...


9

If you are willing to constrain the problem further by letting the network be layered, then Tom Mitchell's "Machine Learning" gives an upper bound of ( $2ds \log(es)$) (section 7.4.4) where $s$ is the number of internal nodes (which must be higher than 2), $d$ is the VC dimension of the individual nodes, and $e$ is the base of the natural logarithm. If you'...


9

Fewer nodes/edges (or edges with fixed weights) means that there are fewer parameters whose values need to be found, and this typically reduces the time to learn. Also, when there are fewer parameters, the space that can be expressed by the neural network has fewer dimensions, so the neural network can only express more general models. It is thus is less ...


9

It seems other answers while informative/ helpful are not actually understanding your question exactly and are reading a little too much into it. You didn't ask if neural networks would outperform other methods, you only asked if they could be applied to NP complete problems. The answer is yes, with some success and this has been known for decades and there ...


8

During the training phase, backpropagation informs each neuron how much it should influence each neuron in the next layer. If the activation function isn't monotonic then increasing the neuron's weight might cause it to have less influence, the opposite of what was intended. The result would be choatic behavior during training, with the network unlikely to ...


7

One very good resource is the Neural Network FAQ. The question depends a lot on your problem. If the problem is linear in nature, there is no reason to have any hidden layers. If the problem is non-linear, often a single hidden layer with around 10 hidden neurons will do the trick. There is a very similar question (with a very similar answer) at ...


7

It depends. Weights of neural networks can be graphed or visualized for some insight. This is especially useful if the neural network works with visual processing. It is possible to "derive" what low-level inputs to the neural network create particular neurons in higher levels to "fire" by working backwards through the neural network weights— in other words, ...


7

With neural networks, you always need to randomly initialize your weights to break symmetry. If you don't use a non-linear activation function in the hidden units, then you might as well have stayed with a single layer. Your network is now just a composition of two linear functions, which is of course just another linear function. That learning rate seems ...


7

For high school kids, I think the most important goal is to make sure they're impressed by what they've accomplished. To do that the task needs to be inherently useful. Classic things like the XOR problem or pixel counting aren't going to do the trick, because you're relying on the students to connect the dots and realize that this means you can build it ...


7

there are many papers on using artificial neural networks to solve TSP including recurrent and Hopfield networks, and they "succeed" in a rough sense, but so far there does not seem to be any evidence that the techniques are in any way (strongly?) superior to other algorithmic approaches, so its something more like a research curiosity at the moment. the use ...


6

I believe what you want to show is that the energy function is monotonically decreasing from time $t$ to time $t+1$ given the state update rules. Since there is only a finite number of states, this means the state must converge to a equilibrium under the given dynamics. To this end let $\mathbf{s}(t) \in \{0,1\}^n$ be the state of the network at time $t$. ...


6

Criticisms against Jeff Hawkins are well summarized in the following essay taken from http://www.theregister.co.uk/2014/03/29/hawkins_ai_feature/ I myself believe that the HTM theory has a huge potential and will be a foundation of true machine intelligence. IBM recently announced to back up the HTM theory and started the Cortical Learning Center including ...


6

Are you able to determine what the algorithm or logic is contained within the neural network? Other than feeding in all possible inputs and studying the outputs it produces. No, I don't think so, not in a meaningful way. That would be akin to studying the bits in each individual byte of a computer program in an effort to evaluate its purpose. You need ...


6

You just add more neurons. You're right that it's often not useful (if you only get 5 bits of information in, it's hard to put 10 bits of information out), but if you want to (e.g. because your output format is less dense), go ahead. As a trivial example, if you wanted to create an ANN to convert characters to graphemes (as represented on an 8x8 grid of on/...


6

From the point of view of filling-out the machine model, the three criteria of Turing Completeness (Böhm-Jacopini theorem) appear useful. Sequence Selection Iteration or Recursion It's clearly #3 that's missing at present, the fanciful "linkage" mentioned above. Edit: This doesn't really help dig me out of the hole with this question, but ... I was ...


6

If you have fixed the weights between the input and hidden units and are only modifying the hidden to output weights during training then there will be no local minima. With fixed input to hidden weights the optimization problem you are solving is similar to logistic regression but with a tanh instead of sigmoid function. Regardless the problem is convex and ...


6

The basic intuition behind initializing weight layers into small (and different) values is just so that the bias of the system is broken and weight values can move along and away and apart to different values. More concretely, you'ld probably want your initial weights to be distinct and have "a small gap" between them, this 'gap' expands out as you go along ...


6

I think you might be confusing the terminology in a way that is making the issue confusing. SVMs work by defining a linear decision boundary, i.e., a hyperplane. We can define this hyperplane in terms of inner products between the points. Therefore, if we define this inner product to be in some high-dimensional, or even infinite dimensional space, what looks ...


6

Many implementations you can find out in the web are done on matrices (MATLAB for instance) since it provides a compact notation. Haykin's textbook on neural networks takes this approach. Matrices also provide a simple translation to hardware design (FPGA, ASIC, etc.). They are also more often implemented on the FPU. If you implement a neural network in an ...


Only top voted, non community-wiki answers of a minimum length are eligible