Skip to main content
42 votes
Accepted

What is Temperature in LSTM (and neural networks generally)?

Temperature is a hyperparameter of LSTMs (and neural networks generally) used to control the randomness of predictions by scaling the logits before applying softmax. For example, in TensorFlow’s ...
Justin Shenk's user avatar
  • 1,035
29 votes

Why can't we say that a Neural Network is a NP problem solver?

SGD is an algorithm, not a problem. It is not NP-complete or NP-hard. SGD is one approach to an optimization problem. Some optimization problems are NP-hard; some are not. All of your deductions ...
D.W.'s user avatar
  • 161k
11 votes

Why can't we say that a Neural Network is a NP problem solver?

Let me start by briefly reviewing the kind of problems under discussion: Problems in P: These are decision problems (where the answer is Yes or No), optimization problems (where the answer is an ...
Yuval Filmus's user avatar
10 votes

Resources for studying the mathematical foundations of machine learning, for someone from a math/physics background

Your assumptions may be a little bit off. Deep learning is largely an engineering field, and it is a young and rapidly moving one. Most "why" questions don't have very good answers. To ...
D.W.'s user avatar
  • 161k
9 votes

Why is the manifold hypothesis true?

Presumably you don't really get a manifold. For one thing, there's probably a nonzero chance of any particular coordinate being perturbed arbitrarily -- an impurity in a crystal, a grey hair on a ...
Daniel McLaury's user avatar
7 votes

Google DeepDream Elaborated

The idea of DeepDream is this: pick some layer from the network (usually a convolutional layer), pass the starting image through the network to extract features at the chosen layer, set the gradient ...
Maxim's user avatar
  • 640
7 votes
Accepted

Traveling Salesman Problem with Neural Network

This Medium post lists the latest (not a full list of course) studies in the combinatorial optimization domain. All three papers use Deep Reinforcement Learning, which does not need any training set ...
tahsin kose's user avatar
7 votes

What's the input to the decoder in a sequence to sequence autoencoder?

I was wondering the same and just stumbled across a nice tutorial by Quoc V. Le. The following explanation deals with the conditional case since this seems to be the common case. My explanation is ...
jns's user avatar
  • 171
6 votes

Is there a "flaw" in the backpropagation algorithm?

There's no paradox. This is not a flaw. Nothing is going wrong here. In both examples, you're making steady progress: the output steadily gets (monotonically) closer and closer to the target. ...
D.W.'s user avatar
  • 161k
6 votes
Accepted

How do neural networks create results like its inputs?

These are known as Autoencoders. As you said, these neural networks are trained to produce output that is similar to the input, rather than output a classification of some kind. Internally, they do ...
Wil's user avatar
  • 368
6 votes
Accepted

Neural Network: Why can't we calculate derivatives during forward prop itself?

You described the simplest case of the neural network, where the center neuron has only one output $a$, which is connected to the final loss function. In general, there can be several outputs and the ...
Maxim's user avatar
  • 640
6 votes
Accepted

Is Artificial General Intelligence possible with our current machine learning models?

The short answer is, we don't know! This is an open question in AI research. We know how neurons transmit signals, and can simulate that in a straightforward way: that's how layered perceptron models ...
Draconis's user avatar
  • 7,138
6 votes

Why is the manifold hypothesis true?

You know what they say about big hands? Big feet! Correlations abound in real high-dimensional data. For instance, if our data set were 206 "length" measurements of human bones, we would definitely ...
Larry B.'s user avatar
  • 193
6 votes

fast and stable x * tanh(log1pexp(x)) computation

With some algebraic manipulation (as pointed out in @orlp's answer), we can deduce the following: $$f(x) = x \tanh(\log(1+e^x)) \tag{1}$$ $$ = x\frac{(1+e^x)^2 - 1}{(1+e^x)^2 + 1} = x\frac{e^{2x} + 2e^...
Yashas's user avatar
  • 275
5 votes
Accepted

What are the limitations of RNNs?

I finally finished the project. Given really short signals and a really small training set, SNNs (I used Echo State Machines and a neural form of SVM) vastly out-performed Deep Learning recurrent ...
Seanny123's user avatar
  • 641
5 votes

Why updating only a part of all neural network weights does not work?

If you're only changing the weights in the last layer, then effectively you have a neural network with a single layer, preceded by some preprocessing step. Single-layer neural networks (also known as ...
Yuval Filmus's user avatar
5 votes

Confused between turing-completeness and universal approximation - are they related?

A CNN can approximate a function on a fixed number of input variables, say $n$ of them. The set of functions on $n$ input variables isn't "Turing-complete". For instance, a boolean function $f:\{0,1\...
D.W.'s user avatar
  • 161k
5 votes
Accepted

How to show that cross entropy is minimized?

You are calculating the so-called binary cross-entropy. Let $f(\cdot)$ be a sigmoid function. The binary cross-entropy between $y$ and $f(t)$ is $$ F(t,y) = H(y,f(t)) = -y\log f(t) - (1-y)\log(1-f(...
user172818's user avatar
5 votes
Accepted

Guided Backpropagation in Deep Neural Networks

Guided backpropagation from the model output to the input of the first ReLU indeed results in non-negative values (as the gradient from ReLU output wrt. ReLU input is set to zero if it is negative). ...
Dominik Friedmann's user avatar
5 votes

Being stuck and frustrated with my masters project

I have been implementing a branch and bound solver with heuristics for an NP-hard problem. It got complicated at some points and had to reimplement parts a couple of times. The problem was (I think), ...
Narek Bojikian's user avatar
5 votes
Accepted

How was the σ function chosen to extend the perceptron?

Answer to first part The function in the question is called the logistic function. Sometimes it is also called "the" sigmoid function, but some authors use sigmoid to just mean any s-shaped ...
Aaron Rotenberg's user avatar
5 votes

Why can't we say that a Neural Network is a NP problem solver?

Non-linear optimization in general is not in NP, it isn't solvable at all. You cannot check in polynomial time whether a solution is the global optimum or just a local one. There is, in fact, no way ...
kutschkem's user avatar
  • 462
5 votes
Accepted

What are common ways to distinguish between local minima and long run-time in hyperparamteer optimization

In general, no. Perhaps here is only one unique input that makes the function produce an enormously negative output; or perhaps there is none. You can't tell the two apart without having examined a ...
D.W.'s user avatar
  • 161k
4 votes
Accepted

error computation in multi layered perceptron

How do we get that $\delta_{ok}$? You calculate the gradient of the network . Have a look at "Tom Mitchel: Machine Learning" if you want to see it in detail. In short, your weight update rule is $$w ...
Martin Thoma's user avatar
  • 2,360
4 votes
Accepted

Is it okay to select the best performers of test cases for scientific publication in neural network machine learning

No. You must use the test set only once. Otherwise your result will not be representative of the true performance of your scheme. That's bad. Instead, use the validation set for checking the ...
D.W.'s user avatar
  • 161k
4 votes
Accepted

Why update weights and biases after training a Neural Network on whole set of training samples

You are right. While you could backpropagate for all samples and then update the weights, you don't have to. Alternatively, you can iterate through the samples and, for each sample, backpropagate ...
D.W.'s user avatar
  • 161k
4 votes
Accepted

Is it possible to add "memory" to a neural network?

Yes it is possible. Recurrent neural networks address this issue. They are networks with loops in them, allowing information to persist.These loops make recurrent neural networks seem kind of ...
Sayali Sonawane's user avatar
4 votes
Accepted

What method of collective recogintion to use for digits recognition?

The state of the art for digit recognition does not use collective recognition, competence areas, ensembles, or any of the other ideas you propose in your question. Instead, the state of the art for ...
D.W.'s user avatar
  • 161k
4 votes
Accepted

Why neural nets are split into layers?

A neural net will work with any arbitrary topology, as long as it has no cycles. We often use layers because it is easy to implement and because it allows a human to set parameters for how large the ...
Lieuwe Vinkhuijzen's user avatar
4 votes
Accepted

Difference Between Residual Neural Net and Recurrent Neural Net?

The answer is YES, they basically are the same according to this paper The figure above shows how they compared both and how a ResNet can be reformulated into a recurrent form tat is almost ...
youssef4's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible