# Tag Info

32

Temperature is a hyperparameter of LSTMs (and neural networks generally) used to control the randomness of predictions by scaling the logits before applying softmax. For example, in TensorFlow’s Magenta implementation of LSTMs, temperature represents how much to divide the logits by before computing the softmax. When the temperature is 1, we compute the ...

29

SGD is an algorithm, not a problem. It is not NP-complete or NP-hard. SGD is one approach to an optimization problem. Some optimization problems are NP-hard; some are not. All of your deductions start from that one incorrect assumption, so they don't actually follow. "Neural Networks" don't have a computational complexity; they aren't a problem.

22

No. This direction is unlikely to be useful, for two reasons: Most computer scientists believe that P $\ne$ NP. Assuming P $\ne$ NP, this means there does not exist any polynomial-time algorithm to solve any NP-complete problem. If you want your neural network to solve the problem in a reasonable amount of time, then it can't be too large, and thus the ...

19

Neural Networks are not magic. If you treat them like they are and just throw data at them without thinking you're going to have a very bad time. You need to stop and ask youself "Is milliseconds since 1970 actually going to be predictive of the event I'm interested in?" The answer you should arrive at immediately is no. Why? For every instance you ...

16

A perceptron is always feedforward, that is, all the arrows are going in the direction of the output. Neural networks in general might have loops, and if so, are often called recurrent networks. A recurrent network is much harder to train than a feedforward network. In addition, it is assumed that in a perceptron, all the arrows are going from layer $i$ ...

14

Artificial neural networks is a class of algorithms that include a lot of different kinds of algorithms based on graphs, so I won't detail here beyond what you asked because there's too much to say, since there are so many kinds of ANNs. The first kind of artificial neural networks, the famous McCulloch-Pitts neurons, were linear, meaning that they could ...

11

Short answer: Strictly speaking, "Deep" and "Spiking" refer to two different aspects of a neural network: "Spiking" refers to the activation of individual neurons, while "Deep" refers to the overall network architecture. Thus in principle there is nothing contradictory about a spiking, deep neural network (in fact, the brain is arguably such a system). ...

11

It seems other answers while informative/ helpful are not actually understanding your question exactly and are reading a little too much into it. You didn't ask if neural networks would outperform other methods, you only asked if they could be applied to NP complete problems. The answer is yes, with some success and this has been known for decades and there ...

11

Let me start by briefly reviewing the kind of problems under discussion: Problems in P: These are decision problems (where the answer is Yes or No), optimization problems (where the answer is an optimal solution to a problem such as minimum spanning tree), or function problems (compute some function of the input) which can be solved efficiently (in a ...

10

artificial neural networks models were generally restricted to only a few layers, say 3, for decades, for various reasons, including a math proof named Kolmogorovs thm that indicated they could theoretically approximate arbitrary functions with low error (but only with many neurons). multilayer networks beyond that were not feasible/effective via prior ...

10

Using Backpropagation with momentum in a network with $n$ different weights $W_k$ the $i$-th correction for weight $W_k$ is given by $\Delta W_k(i) = -\alpha \frac{\partial E}{\partial W_k} + \mu \Delta W_k(i-1)$ where $\frac{\partial E}{\partial W_k}$ is the variation of the loss w.r.t. $W_k$. Introduction of the momentum rate allows the attenuation ...

10

During the training phase, backpropagation informs each neuron how much it should influence each neuron in the next layer. If the activation function isn't monotonic then increasing the neuron's weight might cause it to have less influence, the opposite of what was intended. The result would be choatic behavior during training, with the network unlikely to ...

8

With neural networks, you always need to randomly initialize your weights to break symmetry. If you don't use a non-linear activation function in the hidden units, then you might as well have stayed with a single layer. Your network is now just a composition of two linear functions, which is of course just another linear function. That learning rate seems ...

7

It depends. Weights of neural networks can be graphed or visualized for some insight. This is especially useful if the neural network works with visual processing. It is possible to "derive" what low-level inputs to the neural network create particular neurons in higher levels to "fire" by working backwards through the neural network weights— in other words, ...

7

For high school kids, I think the most important goal is to make sure they're impressed by what they've accomplished. To do that the task needs to be inherently useful. Classic things like the XOR problem or pixel counting aren't going to do the trick, because you're relying on the students to connect the dots and realize that this means you can build it ...

7

Batch learning and online learning both have their place. Generally speaking batch learning will train your neural network to a lower residual error level, because the online training can sometimes have one training undo the effect of another. However, online training has a few advantages: Online learning schemes learn "faster." In some cases, ...

7

there are many papers on using artificial neural networks to solve TSP including recurrent and Hopfield networks, and they "succeed" in a rough sense, but so far there does not seem to be any evidence that the techniques are in any way (strongly?) superior to other algorithmic approaches, so its something more like a research curiosity at the moment. the use ...

7

The idea of DeepDream is this: pick some layer from the network (usually a convolutional layer), pass the starting image through the network to extract features at the chosen layer, set the gradient at that layer equal to the activations themselves, and then backpropagate to the image. Why does it make sense? Intuitively, it amplifies the features that are ...

6

I believe what you want to show is that the energy function is monotonically decreasing from time $t$ to time $t+1$ given the state update rules. Since there is only a finite number of states, this means the state must converge to a equilibrium under the given dynamics. To this end let $\mathbf{s}(t) \in \{0,1\}^n$ be the state of the network at time $t$. ...

6

Criticisms against Jeff Hawkins are well summarized in the following essay taken from http://www.theregister.co.uk/2014/03/29/hawkins_ai_feature/ I myself believe that the HTM theory has a huge potential and will be a foundation of true machine intelligence. IBM recently announced to back up the HTM theory and started the Cortical Learning Center including ...

6

Are you able to determine what the algorithm or logic is contained within the neural network? Other than feeding in all possible inputs and studying the outputs it produces. No, I don't think so, not in a meaningful way. That would be akin to studying the bits in each individual byte of a computer program in an effort to evaluate its purpose. You need ...

6

I think you might be confusing the terminology in a way that is making the issue confusing. SVMs work by defining a linear decision boundary, i.e., a hyperplane. We can define this hyperplane in terms of inner products between the points. Therefore, if we define this inner product to be in some high-dimensional, or even infinite dimensional space, what looks ...

6

The basic intuition behind initializing weight layers into small (and different) values is just so that the bias of the system is broken and weight values can move along and away and apart to different values. More concretely, you'ld probably want your initial weights to be distinct and have "a small gap" between them, this 'gap' expands out as you go along ...

6

Many implementations you can find out in the web are done on matrices (MATLAB for instance) since it provides a compact notation. Haykin's textbook on neural networks takes this approach. Matrices also provide a simple translation to hardware design (FPGA, ASIC, etc.). They are also more often implemented on the FPU. If you implement a neural network in an ...

6

The discount factor does not represent the likelihood to reach the state $s′$from the state $s$. That would be $p(s'|s,a)$, which is not used in Q-Learning, since it is model-free (only model-based reinforcement learning methods use those transition probabilities). The discount factor $γ$ is a hyperparameter tuned by the user which represents how much ...

6

This Medium post lists the latest (not a full list of course) studies in the combinatorial optimization domain. All three papers use Deep Reinforcement Learning, which does not need any training set but learns completely from its own experience. I have been working on the first paper for some time and inference time is on milliseconds level. According to ...

6

These are known as Autoencoders. As you said, these neural networks are trained to produce output that is similar to the input, rather than output a classification of some kind. Internally, they do not differ much from other neural net designs. Simply, the expected output is the input (or a slight variant of the input), rather than a classification. One ...

6

There's no paradox. This is not a flaw. Nothing is going wrong here. In both examples, you're making steady progress: the output steadily gets (monotonically) closer and closer to the target. Looking at individual weights (like w1 or w2), there's no reason to expect them to vary monotonically during the learning process. If we perfect foresight and we ...

6

I was wondering the same and just stumbled across a nice tutorial by Quoc V. Le. The following explanation deals with the conditional case since this seems to be the common case. My explanation is based on and the image is taken from chapter 5 Sequence output prediction with Recurrent Neural Networks. Background We only regard a decoder with a single cell ...

6

You described the simplest case of the neural network, where the center neuron has only one output $a$, which is connected to the final loss function. In general, there can be several outputs and the total error signal coming to this node equals the sum over all output connections. Plus the error messages across these connections are different. And the ...

Only top voted, non community-wiki answers of a minimum length are eligible