Questions tagged [neural-networks]
Network structure inspired by simplified models of biological neurons (brain cells). Neural networks are trained to "learn" by supervised and unsupervised techniques, and can be used to solve optimization problems, approximation problems, classify patterns, and combinations thereof.
125
questions with no upvoted or accepted answers
4
votes
0answers
22 views
How to represent sentences with their dependency parses as input to an RNN?
I am working on a task embedding sentences into a lower-dimensional space according to style, both grammatical and lexical. As such, I want to have as input the linear ordering of tokens in each ...
4
votes
0answers
100 views
Clarification in model precision term for Gaussian Processes
Following the blog post, where the dropout in the deep learning models has been approximated to a Gaussian process. Research paper and its appendix.
Looking at the suggestion of the author from the ...
4
votes
0answers
84 views
What is the practical difference between an neuromorphic processors and GPU's?
IBM is working on a neuromorphic computer, Synapse, intended for computation based on artificial neural networks.
My question is, what makes these types of computers more efficient at processing ...
3
votes
0answers
47 views
When is currying more efficient in deep neural nets?
I'm reading a blog post on deep Q-learning, and it contrasts traditional lookup-table-based Q-learning with deep Q-learning:
What I wonder about in this picture is: Why does the deep NN not also ...
3
votes
0answers
87 views
Was there a phase in Machine Learning timeline when researchers thought some Neural Networks could not be trained?
I was talking to a professor who made a comment to my question.
Me: So much of quality literature around this topic ( IP Protection
for Neural Weights) emanated in 1990-1991, I'm truly at loss ...
3
votes
0answers
601 views
What are the inputs to an LSTM for Slot Filling Task
I am confused on the inputs of a Long-Short Term Memory (LSTM) for the slot filling task in Spoken Language Understanding.
Before I worked on this, I implemented a language model with a Recurrent ...
3
votes
0answers
712 views
Square or circle neural network detection
I am trying to use a simple perceptron to recognize if there is a square or a circle on an image. The images I generated are 300x300 px and I am having issues training the network since the images are ...
3
votes
0answers
70 views
Number of parameters to be optimized in Artificial Bee Colony
I was reading this paper - Software defect prediction using cost-sensitive neural network by Ömer Faruk Arara and Kürsat Ayan
It uses Artificial Bee Colony algorithm to train the neural network. In ...
3
votes
0answers
836 views
The runtime of a neural net with given numbers of observations, features, and neurons
If I have $n$ training observations, $m$ number of features per observation, and my neural network has $x$ neurons in the 1st layer, $y$ neurons in the 2nd layer, and 1 output neuron, what is the ...
3
votes
0answers
439 views
What is the activation function, label and loss function for Hierachical Softmax
Several papers(1 (originator), 2, 3) suggest the use of Hierachical Softmax instead of softmax for classification where the number of classes is large (eg many thousand).
I haven't been able to get ...
3
votes
0answers
88 views
How to select topology for neural network?
I was given a target function to design a neural network and train: $y = (x_1 \wedge x_2) \vee (x_3 \wedge x_4)$
The number of inputs and outputs seems obvious (4 and 1). And the training data can ...
3
votes
0answers
123 views
practical use of a Boltzmann machine
I am reading "Neural Networks and Learning Machines" and in Chapter 11 the book covers Boltzman machines and it is stated "the network [Boltzmann machine] can perform pattern completion", but does not ...
2
votes
0answers
33 views
Efficient top eigenvalue computation for Hessian of neural networks
I train a neural network - one of the Resnet variations ($\approx 10^7$ parameters) on the CIFAR-10 dataset - and after each epoch, I would like to find the smallest/largest eigenvalues of its Hessian....
2
votes
0answers
24 views
Simple back propagation example
Sorry if this is too simplistic of a question, but over the last couple of months I have been working through the course mathematical foundations of machine learning at my college. I think I am really ...
2
votes
0answers
34 views
How do convolution layers work?
So let's say I have a 4x4 image with 3 channels. So we have 3 4x4 matrices where each matrix represents a channel. Now let's say I also have a 3x3 kernel. I know that for convolution layers, I have to ...
2
votes
1answer
97 views
Learning the activation function in a neural network?
Neural networks use specific activation functions, commonly used ones are tanh, ReLu.
I have seen that people have experimented with continuously parametrices activation functions, for instance here. ...
2
votes
0answers
33 views
Multilayer Perceptrons for solving variational problems
Can we use a multilayer perceptron to solve variational problems?
By variational problem I mean something we might encounter in the calculus of variations, for example the geodesic problem: given two ...
2
votes
0answers
101 views
Approximate dot product between neural network output layer's parameter vector and input activations with winner-take-all hashing
In the paper Deep Networks with Large Output Spaces, Vijayanarasimhan et al. describe their approach to approximating the dot product between a neural network's output layer's parameter vector and ...
2
votes
0answers
27 views
How often should I read out information from an echo state recurrent neural network?
Recurrent neural networks makes it possible to implement some kind of memory, which can be very useful for a lot of tasks, incl. (but not limited to) robot control, which I am interested in. For ...
2
votes
0answers
232 views
Multilayer perceptron memory requirements
How much memory do we need to train a multilayer perceptron?
I've started to figure this out myself, but I'm stuck.
I have one-layer MLP. Each training example is a vector of 100 real numbers in ...
2
votes
0answers
115 views
Trying to understand Le's “cat paper”
QV Le et al. show in Building high-level features using large scale unsupervised learning (2012) how to use unsupervised learning of a deep neural net to recognize faces and cats. Working through this ...
2
votes
0answers
35 views
is mappings optimization problems onto neural networks still a thing?
Is mappings optimization problems onto neural networks still a thing? I only found papers are written prior to 1999. These old papers mostly deal with Hopfield network, which i read is obsolete and ...
2
votes
0answers
66 views
How to best model multidimensional, continuous, non-convex “shape” as neural network?
I have:
set of n-dimensional points that I know are inside of the shape
n >= 18, range on all dimensions has upper bound and lower bound (no axis goes to infinity).
shape is pretty large in this n-...
2
votes
0answers
90 views
KDD Machine Learning using K-NN Algorithm Classification Problem
I'm trying to solve a classification problem from the KDD cup archive of 2004.
Details can be found here: KDD 2004 Archive
I'm only dong the particle physics part.
The description of dataset is as ...
2
votes
0answers
12 views
What would happen if a multi-layer perceptron is insufficient to learn a function?
Suppose that something is wrong with its configuration (ie topology), for example there are too few neurons in a certain layer, or not enough layers.
I have an intuition that some neurons' "delta" ($\...
2
votes
0answers
471 views
How to select the maximum weight value for a bias node in a neural network?
I'm programming a neural network. I know that I should initialize the network by picking random weights. How do I pick a random weight for the connections to bias nodes? What distribution should I ...
2
votes
0answers
101 views
How can you use HMMs and ANNs for on-line handwriting recognition?
On-line handwriting recognition is the task of converting a series of $(x(t),y(t))$ coordinates to symbols and words. In contrast to off-line handwriting recognition, where you only have a bitmap of ...
1
vote
1answer
19 views
Binary cross Entropy derivative?
I am just learning backpropagation algorithm for NN and currently I am stuck with the right derivative of Binary Cross Entropy as loss function.
Here it is:
...
1
vote
1answer
31 views
Expressivity of neural networks, how much information can be stored
I want to know whether a given neural network (with a finite number of nodes) is able to store all injective maps f: D -> C, where D has cardinality k and C has cardinality N (so the number of maps ...
1
vote
0answers
37 views
In a machine learning system, why use differentially private SGD if our input data is already perturbed by a DP mechanism?
I'm trying to implement my own version of a deep neural network with differential privacy to preserve the privacy of the parties involved in the training dataset.
I'm using the method by Abadi et al. ...
1
vote
0answers
33 views
How can Kneser-Ney Smoothing be integrated into a neural language model?
I found a paper titled Multimodal representation: Kneser-Ney Smoothing/Skip-Gram based neural language model. I am curious about how the Kneser-Ney Smoothing technique can be integrated into a feed-...
1
vote
0answers
16 views
Why do we need to change the (weight decay) regularization parameter when changing the number of inputs that neural network is being trained with?
I am currently working my way through Michael Nielsen's ebook Neural Networks and Deep Learning and I am reading about overfitting and (L2) regularization. In this subsection, the process of L2 (a.k.a ...
1
vote
0answers
8 views
Is there a way to connect a deep language model output to input?
In models like GPT-2, TXL and Grover, is there a good way to know which input weights (tokens) resulted in each token of the output?
1
vote
0answers
23 views
Value flow (and economics) in stacked reinforcement learning systems: agent as reinforcement environment for other agents?
There is evolving notion of stacked reinforcement learning systems, e.g. https://www.ijcai.org/proceedings/2018/0103.pdf - where one RL systems executes actions of the second RL system and it itself ...
1
vote
1answer
148 views
What does “Temporal extent” mean?
I am reading Long-term Temporal Convolutions for Action Recognition and under the Section 3.1, I read this:
To investigate the impact of long-term temporal convolutions, we here study network inputs ...
1
vote
0answers
135 views
Converting (reverse-engineering) Turing machine into program or most concise algorithm?
It is known that every program or every algorithm can be converted to Turing machine. But what about the reverse process? Is there algorithm (or research trend that considers such algorithm) to ...
1
vote
0answers
411 views
Why is the method of im2col with GEMM is more efficient than the method of direction implementation with SIMD in CNN
The convolutional layers are most computationally intense parts of Convolutional neural networks (CNNs).Currently the common approach to impement
convolutional layers is to expand the image into a ...
1
vote
0answers
149 views
Proof of perceptron convergence theorem for ZERO threshold?
The generalized perceptron convergence theorem is for a defined threshold T.
When you do the maths it all comes to an upper bound and a lower bound.
The lower bound looks like this!
Therefore
...
1
vote
0answers
350 views
How can node2vec help find similar “roles” within a graph (nodes whose connections have similar structure within the graph)?
I have a question on the node2vec algorithm described in this paper.
Node2vec is a deep learning algorithm that word2vec to graphs to learn embeddings. The authors claim that it can help find nodes ...
1
vote
0answers
28 views
Classify manifolds with neural networks
Can a neural network be used to find the genus of a 2-manifold given for instance as a CW complex?
1
vote
0answers
33 views
How would you go about creating a algorithm that should generate a shakespearean sonnet on any given theme
I need to create an algorithm that is going to create a shakespearean sonnet for a specific theme. This theme should be generated out of twitter tweets that have some hashtag.
My current idea goes ...
1
vote
0answers
615 views
Backward mapping with bilinear sampler
I have some experiences with Convolutional Neural Networks before. I have a question regarding the Bilinear Sampler used in "Unsupervised Monocular Depth Estimation With Left-Right Consistency" (the ...
1
vote
0answers
40 views
Does a deep feedforward neural net approximate a (single or multivariate)polynomial?
Does composition of several linear transformation plus non-linear activation function in each layer and different layers (as they are in feedforward neural net) represent or approximate a ...
1
vote
0answers
159 views
Recurrent neural networks (Hopfield-like) with short limit cycles
Standard Hopfield networks exhibit stable patterns (states) which are attractors of a dynamic system. I wonder how to modify standard Hopfield networks such that they exhibit stable limit cycles as ...
1
vote
0answers
22 views
How do RNN's map variable-length sequences to variable-length sequences?
According to Karpathy's blog "The Unreasonable Effectiveness of Recurrent Neural Networks", recurrent neural networks can map variable-length sequences to variable-length sequences, as shown by the ...
1
vote
0answers
201 views
Distributing Neural Network Training World-Wide Using Blockchain?
I just finished reading a white-paper from a recent AI startup. The company, Deep Brain Chain, wants to distribute neural network training over computers worldwide, using blockchain technology. Here ...
1
vote
0answers
119 views
adjusted fitness in NEAT algorithm
I'm learning about NEAT from the following paper: http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf
I'm having trouble understanding how adjusted fitness penalizes large species and prevents ...
1
vote
0answers
49 views
Detecting shapes using CNNs in respect to other shapes in the image
I'm solving a problem of detecting an optic disc in a retina image. As you can see from the image:
the optic disc is the epicentrum of the blood vessels, has an irregular circular shape and has a ...
1
vote
0answers
152 views
Can CNN be used for “unknown correlated” data?
Now the title may be misleading, but I don't know how to give it a better name.
My question is if I have data in a 2d matrix and under the assumption that:
-I know for sure that there are ...
1
vote
0answers
50 views
Finding a line shape objects (bars) in video images using neural networks (instead of Hough transform)
I'm currently trying to learn about computer vision stuff and I want to try it on a simple project of real time tracking of figures of players in a foosball game while the camera can be moving a ...
