42
votes
Accepted
What is Temperature in LSTM (and neural networks generally)?
Temperature is a hyperparameter of LSTMs (and neural networks generally) used to control the randomness of predictions by scaling the logits before applying softmax. For example, in TensorFlow’s ...
- 1,035
29
votes
Why can't we say that a Neural Network is a NP problem solver?
SGD is an algorithm, not a problem. It is not NP-complete or NP-hard. SGD is one approach to an optimization problem. Some optimization problems are NP-hard; some are not.
All of your deductions ...
D.W.♦
- 164k
11
votes
Why can't we say that a Neural Network is a NP problem solver?
Let me start by briefly reviewing the kind of problems under discussion:
Problems in P: These are decision problems (where the answer is Yes or No), optimization problems (where the answer is an ...
- 278k
10
votes
Resources for studying the mathematical foundations of machine learning, for someone from a math/physics background
Your assumptions may be a little bit off. Deep learning is largely an engineering field, and it is a young and rapidly moving one. Most "why" questions don't have very good answers. To ...
D.W.♦
- 164k
9
votes
Why is the manifold hypothesis true?
Presumably you don't really get a manifold.
For one thing, there's probably a nonzero chance of any particular coordinate being perturbed arbitrarily -- an impurity in a crystal, a grey hair on a ...
- 762
7
votes
Google DeepDream Elaborated
The idea of DeepDream is this: pick some layer from the network (usually a convolutional layer), pass the starting image through the network to extract features at the chosen layer, set the gradient ...
- 640
7
votes
Accepted
Traveling Salesman Problem with Neural Network
This Medium post lists the latest (not a full list of course) studies in the combinatorial optimization domain. All three papers use Deep Reinforcement Learning, which does not need any training set ...
- 86
7
votes
What's the input to the decoder in a sequence to sequence autoencoder?
I was wondering the same and just stumbled across a nice tutorial by Quoc V. Le. The following explanation deals with the conditional case since this seems to be the common case. My explanation is ...
- 171
6
votes
Accepted
How do neural networks create results like its inputs?
These are known as Autoencoders. As you said, these neural networks are trained to produce output that is similar to the input, rather than output a classification of some kind.
Internally, they do ...
- 368
6
votes
Accepted
Neural Network: Why can't we calculate derivatives during forward prop itself?
You described the simplest case of the neural network, where the center neuron has only one output $a$, which is connected to the final loss function. In general, there can be several outputs and the ...
- 640
6
votes
Accepted
Is Artificial General Intelligence possible with our current machine learning models?
The short answer is, we don't know! This is an open question in AI research.
We know how neurons transmit signals, and can simulate that in a straightforward way: that's how layered perceptron models ...
- 7,176
6
votes
Why is the manifold hypothesis true?
You know what they say about big hands? Big feet!
Correlations abound in real high-dimensional data. For instance, if our data set were 206 "length" measurements of human bones, we would definitely ...
- 193
6
votes
fast and stable x * tanh(log1pexp(x)) computation
With some algebraic manipulation (as pointed out in @orlp's answer), we can deduce the following:
$$f(x) = x \tanh(\log(1+e^x)) \tag{1}$$
$$ = x\frac{(1+e^x)^2 - 1}{(1+e^x)^2 + 1} = x\frac{e^{2x} + 2e^...
- 275
5
votes
Accepted
What are the limitations of RNNs?
I finally finished the project. Given really short signals and a really small training set, SNNs (I used Echo State Machines and a neural form of SVM) vastly out-performed Deep Learning recurrent ...
- 641
5
votes
Why updating only a part of all neural network weights does not work?
If you're only changing the weights in the last layer, then effectively you have a neural network with a single layer, preceded by some preprocessing step. Single-layer neural networks (also known as ...
- 278k
5
votes
Confused between turing-completeness and universal approximation - are they related?
A CNN can approximate a function on a fixed number of input variables, say $n$ of them. The set of functions on $n$ input variables isn't "Turing-complete". For instance, a boolean function $f:\{0,1\...
D.W.♦
- 164k
5
votes
Accepted
How to show that cross entropy is minimized?
You are calculating the so-called binary cross-entropy. Let $f(\cdot)$ be a sigmoid function. The binary cross-entropy between $y$ and $f(t)$ is
$$ F(t,y) = H(y,f(t)) = -y\log f(t) - (1-y)\log(1-f(...
- 434
5
votes
Accepted
Guided Backpropagation in Deep Neural Networks
Guided backpropagation from the model output to the input of the first ReLU indeed results in non-negative values (as the gradient from ReLU output wrt. ReLU input is set to zero if it is negative). ...
5
votes
Being stuck and frustrated with my masters project
I have been implementing a branch and bound solver with heuristics for an NP-hard problem. It got complicated at some points and had to reimplement parts a couple of times. The problem was (I think), ...
- 4,484
5
votes
Accepted
How was the σ function chosen to extend the perceptron?
Answer to first part
The function in the question is called the logistic function. Sometimes it is also called "the" sigmoid function, but some authors use sigmoid to just mean any s-shaped ...
- 3,553
5
votes
Why can't we say that a Neural Network is a NP problem solver?
Non-linear optimization in general is not in NP, it isn't solvable at all. You cannot check in polynomial time whether a solution is the global optimum or just a local one. There is, in fact, no way ...
- 480
5
votes
Accepted
What are common ways to distinguish between local minima and long run-time in hyperparamteer optimization
In general, no. Perhaps here is only one unique input that makes the function produce an enormously negative output; or perhaps there is none. You can't tell the two apart without having examined a ...
D.W.♦
- 164k
4
votes
Accepted
Why update weights and biases after training a Neural Network on whole set of training samples
You are right. While you could backpropagate for all samples and then update the weights, you don't have to. Alternatively, you can iterate through the samples and, for each sample, backpropagate ...
D.W.♦
- 164k
4
votes
Accepted
Is it possible to add "memory" to a neural network?
Yes it is possible.
Recurrent neural networks address this issue. They are networks with loops in them, allowing information to persist.These loops make recurrent neural networks seem kind of ...
- 156
4
votes
Accepted
What method of collective recogintion to use for digits recognition?
The state of the art for digit recognition does not use collective recognition, competence areas, ensembles, or any of the other ideas you propose in your question.
Instead, the state of the art for ...
D.W.♦
- 164k
4
votes
Accepted
Why neural nets are split into layers?
A neural net will work with any arbitrary topology, as long as it has no cycles. We often use layers because it is easy to implement and because it allows a human to set parameters for how large the ...
- 4,437
4
votes
Accepted
Difference Between Residual Neural Net and Recurrent Neural Net?
The answer is YES, they basically are the same according to this paper
The figure above shows how they compared both and how a ResNet can be reformulated into a recurrent form tat is almost ...
- 56
4
votes
Accepted
How can image processing neural networks be effectively trained?
Yes. Typically a very large data set is required: tens or hundreds of thousands of labelled images, or perhaps even more. No, 100 images almost certainly won't be enough.
Just running over the same ...
D.W.♦
- 164k
4
votes
Why do people insist to use the term "multilayer perceptron" instead of "multilayer perceptron network"?
Personally, I've seen "multilayer neural network" used more commonly than "multilayer perceptron". If you like the former term better, use it.
That said: multilayer perceptron is an accepted term of ...
D.W.♦
- 164k
4
votes
Accepted
Measuring difference between two sets of neural network weights?
One way to compare two neural networks is to compare how similar their predictions are, on typical instances.
Ideally, we'd like to compute the expected value of this similarity, taken over the ...
D.W.♦
- 164k
Only top scored, non community-wiki answers of a minimum length are eligible