Questions tagged [neural-networks]

Network structure inspired by simplified models of biological neurons (brain cells). Neural networks are trained to "learn" by supervised and unsupervised techniques, and can be used to solve optimization problems, approximation problems, classify patterns, and combinations thereof.

Filter by
Sorted by
Tagged with
28
votes
2answers
652 views

Why do neural networks seem to perform better with restrictions placed on their topology?

Fully connected (at least layer to layer with more than 2 hidden layers) backprop networks are universal learners. Unfortunately, they are often slow to learn and tend to over-fit or have awkward ...
23
votes
1answer
12k views

What is Temperature in LSTM (and neural networks generally)?

One of the hyperparameters for LSTM networks is temperature. What is it?
22
votes
4answers
13k views

What is the difference between a Neural Network, a Deep Learning System and a Deep Belief Network?

What is the difference between a Neural Network, a Deep Learning System and a Deep Belief Network? As I recall your basic neural network is a 3 layers kinda thing, and I have had Deep Belief Systems ...
17
votes
1answer
766 views

Efficiently computing or approximating the VC-dimension of a neural network

My goal is to solve the following problem, which I have described by its input and output: Input: A directed acyclic graph $G$ with $m$ nodes, $n$ sources, and $1$ sink ($m > n \geq 1$). Output: ...
16
votes
2answers
9k views

Must Neural Networks always converge?

Introduction Step One I wrote a standard backpropegating neural network, and to test it, I decided to have it map XOR. It is a 2-2-1 network (with tanh activation function) ...
16
votes
2answers
8k views

What are the criticisms regarding the performance of HTM?

I just recently learned about the existence of this hierarchical temporal memory (HTM). I already read the document Hierarchical Temporal Memory: Concepts, Theory and Terminology (by Jeff Hawkins and ...
13
votes
3answers
17k views

what is difference between multilayer perceptron and multilayer neural network?

When do we say that a artificial neural network is a multilayer Perceptron? And when do we say that a artificial neural network is a multilayer? Is the term perceptron related to learning rule to ...
12
votes
2answers
10k views

How to encode date as input in neural network?

I am using neural networks to predict a time series. The question I'm facing now is how do I encode date/time/serial no. of each input set as an input to the neural network? Should I use 1 of C ...
11
votes
1answer
213 views

Google DeepDream Elaborated

I've seen a few questions on this site about Deep Dream, however none of them seem to actually speak as to what DeepDream is doing, specifically. As far as I've gathered, they seem to have changed the ...
10
votes
3answers
4k views

Evolving artificial neural networks for solving NP problems

I've recently read a really interesting blog entry from Google Research Blog talking about neural network. Basically they use this neural networks for solving various problems like image recognition. ...
10
votes
2answers
1k views

Is computational power of Neural networks related to the activation function

It is proven that neural networks with rational weights has the computational power of the Universal Turing Machine Turing computability with Neural Nets. From what I get, it seems that using real-...
10
votes
1answer
1k views

How many layers should a neural network have?

Are there any advantages of having more than 2 hidden layers in a Neural Network? I've seen some places that recommend it, others prove that there is no advantage. Which one is right?
10
votes
1answer
6k views

The meaning of discount factor on reinforcement learning

After reading of the google deepmind achievements on Atari's games, I am trying to understand the q-learning and q-networks, but I am little bit confused. The confusion arise in the concept of the ...
9
votes
1answer
5k views

Could an artificial neural network algorithm be expressed in terms of map-reduce operations?

Could an artificial neural network algorithm be expressed in terms of map-reduce operations? I am also interested more generally in methods of parallelization as applied to ANNs and their application ...
9
votes
1answer
23k views

How does the momentum term for backpropagation algorithm work?

When updating the weights of a neural network using the backpropagation algorithm with a momentum term, should the learning rate be applied to the momentum term as well? Most of the information I ...
9
votes
3answers
1k views

When should I move beyond k nearest neighbour

For many machine learning projects that we do, we start with the k Nearest Neighbour classifier. This is an ideal starting classifier as we usually have sufficient time to calculate all distances and ...
9
votes
3answers
3k views

Kernelization trick, for neural networks

I've been learning about neural networks and SVMs. The tutorials I've read have emphasized how important kernelization is, for SVMs. Without a kernel function, SVMs are just a linear classifier. ...
9
votes
1answer
2k views

Why are weights of Neural Networks initialized with random numbers?

Why are neural networks initial weights initialized as random numbers? I had read somewhere that this is done to "break the symmetry" and this makes the neural network learn faster. How does breaking ...
8
votes
2answers
3k views

What can be learned from the weights in a neural network?

I'm very new to neural networks, and have been trying to figure some things out. So, let's say you come across a neural network which has 100 inputs, a hidden layer with 200 nodes, and 32 outputs. ...
8
votes
1answer
2k views

Should activation function be monotonic in neural networks?

A lot of activation functions in neural networks (sigmoid, tanh, softmax) are monotonic, continuous and differentiable (except of may be a couple of points, where derivative does not exist). I ...
8
votes
1answer
5k views

Neural network diverging instead of converging

I have implemented a neural network (using CUDA) with 2 layers. (2 Neurons per layer). I'm trying to make it learn 2 simple quadratic polynomial functions using backpropagation. But instead of ...
8
votes
1answer
750 views

Adapting neural network

I have on a few occasions trained neural networks (back propagation networks) with some rather complicated data sets (backgammon positions and OCR). When doing this, it seems that a lot of the work ...
8
votes
1answer
2k views

Is there a simple way of calculating perceptron weights based on a classification graph?

I am studying for an AI exam and I'm looking for a better way of solving the following problem: Graph shows a classification problem in the unit square $[0,1]^2$, where Class A is denoted by the ...
8
votes
1answer
291 views

The essential difference between spiking neural networks and earlier generation ANN's

I have been studying Spiking Neural Networks online from various papers, mainly Maass (1997). I am not entirely sure I understand what makes SNN's pulse-code in contrast to earlier ANNs which are ...
7
votes
2answers
14k views

What are the key differences between Spiking Neural Network and Deep Learning

Deep Learning, now one of the most popular fields in Artificial Neural Network, has shown great promise in terms of its accuracies on data sets. How does it compare to Spiking Neural Network. Recently ...
7
votes
4answers
9k views

Traveling Salesman Problem with Neural Network

I was curious if there were any new developments in solving the traveling salesman problem using something like a Hopfield recurrent neural network. I feel like I saw something about recent research ...
7
votes
1answer
677 views

Intuitive description for training of LSTM (with forget gate/peephole)?

I am a CS undergraduate (but I don't know much about AI though, did not take any courses on it, and definitely nothing about NN until recently) who is about to do a school project in AI, so I pick a ...
6
votes
3answers
816 views

What piece am I missing to turn this idea into a programming language?

I've been doing some reading (I'll name drop along the way) and have selected a few scattered ideas that I think could be cobbled together into a nifty esoteric programming language. But I'm having ...
6
votes
2answers
7k views

Convolutional Neural Network Example in Tensorflow

I want to ask the dimension change in different convolution and max-pooling layer. I am referring to the example in TensorFlow tutorial: http://tensorflow.org/tutorials/mnist/pros/index.html#deep-...
6
votes
1answer
114 views

Neural Network: Why can't we calculate derivatives during forward prop itself?

I have been studying Andrew Ng's Coursera course about Deep Learning. In that he mentions that we calculate the activation functions during the forward pass, and the derivatives $\dfrac{dL}{dz} \ $, $\...
6
votes
2answers
1k views

Derivation of the energy function of a hopfield network

Can someone please point me towards a rigorous derivation of the energy function of a discrete Hopfield network. What I want, is the derivation must start out with the structure of the network and ...
6
votes
1answer
942 views

How would a neural network deal with an arbitrary length output?

I've been looking into Recurrent Neural Networks, but I don't understand what the architecture of a neural network would look like when the output length is not necessarily fixed. It seems like most ...
6
votes
1answer
172 views

The theory behind backpropogation, gradient calculation

I am reading a book about Neural Networks, that explains the backpropagation algorithm. It explains that the error function is the sum of error of all inputs, and that the algorithm minimizes this sum,...
6
votes
1answer
112 views

What is the role of Numerical Gradient Computation in Backpropagation algorithm?

I was listening CS231n (2017) lectures and noted that there is a lot of attention to Numerical Gradient Computation (NGC). It starts @5:53 in this video and appears a few times later. Also, looking ...
6
votes
2answers
157 views

How can an artificial neural net change the sign of a weight?

My neural net is having trouble switching the sign of a weight. The issue is that the deltas applied to the weight are proportional to that weight, so when it gets closer to zero, the deltas become ...
6
votes
2answers
5k views

How do you determine the inputs to a neural network?

I'm looking at this tutorial on neural networks. The data that is given from the UCI study includes various attributes, such as "mean x of on pixels", "total # on pixels" etc, which are taken as input ...
6
votes
1answer
1k views

How exactly do you calculate the hidden layer gradients in the backpropagation algorithm?

I have been going through the description of the backpropagation algorithm found here. and I am having a bit of trouble getting my head around some of the linear algebra. Say I have a final output ...
6
votes
1answer
2k views

Train a neural network to play tic tac toe using a genetic algorithm

I have an assignment for school, in which I have to build a neural network that will play tic tac toe, using genetic algorithms for training. The thing is that I am clueless on how to connect the two. ...
6
votes
1answer
5k views

Does there exist a data compression algorithm that uses a large dataset distributed with the encoder/decoder?

If my goal were to compress say 10,000 images and I could include a dictionary or some sort of common database that the compressed data for each image would reference, could I use a large dictionary ...
5
votes
3answers
862 views

Is it possible to implement a Neural Network using a graph data structure?

I'm trying to implement a feedforward neural network using a graph. The thing is: I haven't found any example in which is used a graph data structure. So far the examples I've found used arrays. Can ...
5
votes
1answer
729 views

Is there a “flaw” in the backpropagation algorithm?

While trying to find a better backpropagation algorithm, I came across a paradox in my algorithm and then I found out this also happens in the usual backpropagation algorithm. Our neural network ...
5
votes
1answer
2k views

What's the input to the decoder in a sequence to sequence autoencoder?

What's the input to the decoder part of a sequence to sequence autoencoder? I've seen certain examples of such an autoencoder (using LSTM's more often than not) but am still unclear. For example, ...
5
votes
1answer
233 views

Something I don't understand about Genetic Algorithms

I've had a bit of experience programming Neural networks but I am fairly new with genetic algorithms (I'm only 17). I have a major issue that I can't understand. If a child get's one chromatid from ...
5
votes
1answer
457 views

Confused between turing-completeness and universal approximation - are they related?

I am trying to de-knot a point of confusion in my mind regarding "turing-completeness" and the "universal approximation theorem". The context here is deep neural nets: So, consider two types of ...
5
votes
1answer
258 views

Google Deep Dream has these understandings?

From both my own exploration of Google Deep Dream using Dreamify for IOS, and from Google Image results on the topic. I've come to 3 conclusions about the networks understanding of images that seem ...
5
votes
2answers
588 views

What exactly representational bottleneck in InceptionV3 means?

I am trying to understand the concepts behind the InceptionNet V3 and got confused with the meaning of representational bottleneck. They said One should avoid bottlenecks with extreme compression. ...
5
votes
3answers
326 views

Are the Confabulation Theories of Thaler and Hecht-Nielsen Isomorphic?

Both S. L. Thaler and R. Hecht-Nielsen have set forth neural-based theories of "confabulation" applicable to machine learning. The essential mathematics of Hecht-Nielsen is set forth in his paper "...
5
votes
1answer
163 views

Flaw with Cross Entropy Error in Neural Networks

I've recently been working on creating a neural network to classify handwritten digits. I implemented 1-of-N encoding such that there are the same number of output nodes as possible digits (The ...
5
votes
1answer
188 views

Why does the effectiveness of my reinforcement based neural network recede after a while?

I have a reinforcement based neural network training on the OpenAI gym CartPole-v1 environment. For the structure and training algorithm, assume it is the same as the one in this article. Typically, ...
5
votes
1answer
166 views

Predicting next action to take to reach a final state

Does anyone know of an algorithm that could be used to determine the next action to take to reach a desired state when trained on time-series data? For example, a robot starts at a certain state, ...