38 votes
Accepted

What is Temperature in LSTM (and neural networks generally)?

Temperature is a hyperparameter of LSTMs (and neural networks generally) used to control the randomness of predictions by scaling the logits before applying softmax. For example, in TensorFlow’s ...
user avatar
29 votes

Why can't we say that a Neural Network is a NP problem solver?

SGD is an algorithm, not a problem. It is not NP-complete or NP-hard. SGD is one approach to an optimization problem. Some optimization problems are NP-hard; some are not. All of your deductions ...
user avatar
  • 143k
22 votes
Accepted

Evolving artificial neural networks for solving NP problems

No. This direction is unlikely to be useful, for two reasons: Most computer scientists believe that P $\ne$ NP. Assuming P $\ne$ NP, this means there does not exist any polynomial-time algorithm to ...
user avatar
  • 143k
18 votes
Accepted

what is difference between multilayer perceptron and multilayer neural network?

A perceptron is always feedforward, that is, all the arrows are going in the direction of the output. Neural networks in general might have loops, and if so, are often called recurrent networks. A ...
user avatar
  • 13.3k
14 votes

What is the difference between a Neural Network, a Deep Learning System and a Deep Belief Network?

Artificial neural networks is a class of algorithms that include a lot of different kinds of algorithms based on graphs, so I won't detail here beyond what you asked because there's too much to say, ...
user avatar
  • 346
11 votes
Accepted

Should activation function be monotonic in neural networks?

During the training phase, backpropagation informs each neuron how much it should influence each neuron in the next layer. If the activation function isn't monotonic then increasing the neuron's ...
user avatar
  • 7,863
11 votes

Evolving artificial neural networks for solving NP problems

It seems other answers while informative/ helpful are not actually understanding your question exactly and are reading a little too much into it. You didn't ask if neural networks would outperform ...
user avatar
  • 10.8k
11 votes

Why can't we say that a Neural Network is a NP problem solver?

Let me start by briefly reviewing the kind of problems under discussion: Problems in P: These are decision problems (where the answer is Yes or No), optimization problems (where the answer is an ...
user avatar
10 votes
Accepted

How does the momentum term for backpropagation algorithm work?

Using Backpropagation with momentum in a network with $n$ different weights $W_k$ the $i$-th correction for weight $W_k$ is given by $\Delta W_k(i) = -\alpha \frac{\partial E}{\partial W_k} + \mu \...
user avatar
  • 216
9 votes

Why is the manifold hypothesis true?

Presumably you don't really get a manifold. For one thing, there's probably a nonzero chance of any particular coordinate being perturbed arbitrarily -- an impurity in a crystal, a grey hair on a ...
user avatar
8 votes

What are the advantages of online learning when training neural networks?

Batch learning and online learning both have their place. Generally speaking batch learning will train your neural network to a lower residual error level, because the online training can sometimes ...
user avatar
  • 3,085
7 votes

Machine Learning and Neural Networks for High School Students

For high school kids, I think the most important goal is to make sure they're impressed by what they've accomplished. To do that the task needs to be inherently useful. Classic things like the XOR ...
user avatar
  • 1,475
7 votes

Traveling Salesman Problem with Neural Network

there are many papers on using artificial neural networks to solve TSP including recurrent and Hopfield networks, and they "succeed" in a rough sense, but so far there does not seem to be any evidence ...
user avatar
  • 10.8k
7 votes

Google DeepDream Elaborated

The idea of DeepDream is this: pick some layer from the network (usually a convolutional layer), pass the starting image through the network to extract features at the chosen layer, set the gradient ...
user avatar
  • 640
6 votes

What are the criticisms regarding the performance of HTM?

Criticisms against Jeff Hawkins are well summarized in the following essay taken from http://www.theregister.co.uk/2014/03/29/hawkins_ai_feature/ I myself believe that the HTM theory has a huge ...
user avatar
6 votes
Accepted

Is it possible to implement a Neural Network using a graph data structure?

Many implementations you can find out in the web are done on matrices (MATLAB for instance) since it provides a compact notation. Haykin's textbook on neural networks takes this approach. Matrices ...
user avatar
  • 1,584
6 votes

The meaning of discount factor on reinforcement learning

The discount factor does not represent the likelihood to reach the state $s′ $from the state $s$. That would be $p(s'|s,a)$, which is not used in Q-Learning, since it is model-free (only model-based ...
user avatar
  • 481
6 votes
Accepted

Traveling Salesman Problem with Neural Network

This Medium post lists the latest (not a full list of course) studies in the combinatorial optimization domain. All three papers use Deep Reinforcement Learning, which does not need any training set ...
user avatar
6 votes
Accepted

How do neural networks create results like its inputs?

These are known as Autoencoders. As you said, these neural networks are trained to produce output that is similar to the input, rather than output a classification of some kind. Internally, they do ...
user avatar
6 votes

Is there a "flaw" in the backpropagation algorithm?

There's no paradox. This is not a flaw. Nothing is going wrong here. In both examples, you're making steady progress: the output steadily gets (monotonically) closer and closer to the target. ...
user avatar
  • 143k
6 votes

What's the input to the decoder in a sequence to sequence autoencoder?

I was wondering the same and just stumbled across a nice tutorial by Quoc V. Le. The following explanation deals with the conditional case since this seems to be the common case. My explanation is ...
user avatar
  • 161
6 votes
Accepted

Neural Network: Why can't we calculate derivatives during forward prop itself?

You described the simplest case of the neural network, where the center neuron has only one output $a$, which is connected to the final loss function. In general, there can be several outputs and the ...
user avatar
  • 640
6 votes

Why is the manifold hypothesis true?

You know what they say about big hands? Big feet! Correlations abound in real high-dimensional data. For instance, if our data set were 206 "length" measurements of human bones, we would definitely ...
user avatar
  • 193
6 votes

fast and stable x * tanh(log1pexp(x)) computation

With some algebraic manipulation (as pointed out in @orlp's answer), we can deduce the following: $$f(x) = x \tanh(\log(1+e^x)) \tag{1}$$ $$ = x\frac{(1+e^x)^2 - 1}{(1+e^x)^2 + 1} = x\frac{e^{2x} + 2e^...
user avatar
  • 265
5 votes

Machine Learning and Neural Networks for High School Students

NNs are fairly complicated to understand. Your class might be better off starting with a simpler ML algorithm, like a Monte Carlo simulation of a genetic algorithm. Virtually any optimization problem ...
user avatar
5 votes
Accepted

Closed form solution for a single layer linear perceptron

Yes, there is a closed form solution. In the most general terms, $WX = Y$ is a linear equation, so it can be solved as $W = X^{-1}Y$. If $X$ has no inverse, using the pseudo-inverse $X^\dagger = X^T(...
user avatar
  • 1,475
5 votes

Machine Learning and Neural Networks for High School Students

For the second question, I think it is interesting to do some work on face recognition from Tom Mitchell's Machine Learning. (There are available code framework and image data for downloading) The ...
user avatar
  • 101
5 votes

What are the key differences between Spiking Neural Network and Deep Learning

The largest application of SNNs that I know of is Spaun, whose neural networks were built using the Neural Engineering Framework and the Nengo neural simulator. The distinction between SNNs and Deep ...
user avatar
  • 641
5 votes
Accepted

rationale behind multi-layer networks

The rationale is that the number of neurones you need to use for approximating any given contiguous function decreases as you add subsequent layers. See Colah's Deep Learning, NLP, and Representations ...
user avatar
  • 168
5 votes
Accepted

The theory behind backpropogation, gradient calculation

The gradient, just like the derivative, is linear: for functions $f,g$ on the same variables and scalars $\alpha,\beta$: $$ \nabla(\alpha f + \beta g) = \alpha \nabla f + \beta \nabla g. $$
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible