Questions tagged [neural-networks]

Network structure inspired by simplified models of biological neurons (brain cells). Neural networks are trained to "learn" by supervised and unsupervised techniques, and can be used to solve optimization problems, approximation problems, classify patterns, and combinations thereof.

137 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
59 views

Universal approximation bounds of the form $\|f(x)-\hat{f}(x;w)\|\leq \varepsilon \|f(x)\|$

It is known that for every $\varepsilon>0$ there is an appropriate neural network architecture, such that one can approximate any continuous function $f:[0,1]^n\to[0,1]^m$ by the neural network ...
4
votes
0answers
22 views

How to represent sentences with their dependency parses as input to an RNN?

I am working on a task embedding sentences into a lower-dimensional space according to style, both grammatical and lexical. As such, I want to have as input the linear ordering of tokens in each ...
4
votes
0answers
102 views

Clarification in model precision term for Gaussian Processes

Following the blog post, where the dropout in the deep learning models has been approximated to a Gaussian process. Research paper and its appendix. Looking at the suggestion of the author from the ...
4
votes
0answers
88 views

What is the practical difference between an neuromorphic processors and GPU's?

IBM is working on a neuromorphic computer, Synapse, intended for computation based on artificial neural networks. My question is, what makes these types of computers more efficient at processing ...
3
votes
0answers
50 views

When is currying more efficient in deep neural nets?

I'm reading a blog post on deep Q-learning, and it contrasts traditional lookup-table-based Q-learning with deep Q-learning: What I wonder about in this picture is: Why does the deep NN not also ...
3
votes
0answers
89 views

Was there a phase in Machine Learning timeline when researchers thought some Neural Networks could not be trained?

I was talking to a professor who made a comment to my question. Me: So much of quality literature around this topic ( IP Protection for Neural Weights) emanated in 1990-1991, I'm truly at loss ...
3
votes
0answers
617 views

What are the inputs to an LSTM for Slot Filling Task

I am confused on the inputs of a Long-Short Term Memory (LSTM) for the slot filling task in Spoken Language Understanding. Before I worked on this, I implemented a language model with a Recurrent ...
3
votes
0answers
733 views

Square or circle neural network detection

I am trying to use a simple perceptron to recognize if there is a square or a circle on an image. The images I generated are 300x300 px and I am having issues training the network since the images are ...
3
votes
0answers
70 views

Number of parameters to be optimized in Artificial Bee Colony

I was reading this paper - Software defect prediction using cost-sensitive neural network by Ömer Faruk Arara and Kürsat Ayan It uses Artificial Bee Colony algorithm to train the neural network. In ...
3
votes
0answers
879 views

The runtime of a neural net with given numbers of observations, features, and neurons

If I have $n$ training observations, $m$ number of features per observation, and my neural network has $x$ neurons in the 1st layer, $y$ neurons in the 2nd layer, and 1 output neuron, what is the ...
3
votes
0answers
451 views

What is the activation function, label and loss function for Hierachical Softmax

Several papers(1 (originator), 2, 3) suggest the use of Hierachical Softmax instead of softmax for classification where the number of classes is large (eg many thousand). I haven't been able to get ...
3
votes
0answers
92 views

How to select topology for neural network?

I was given a target function to design a neural network and train: $y = (x_1 \wedge x_2) \vee (x_3 \wedge x_4)$ The number of inputs and outputs seems obvious (4 and 1). And the training data can ...
3
votes
0answers
124 views

practical use of a Boltzmann machine

I am reading "Neural Networks and Learning Machines" and in Chapter 11 the book covers Boltzman machines and it is stated "the network [Boltzmann machine] can perform pattern completion", but does not ...
2
votes
0answers
69 views

Efficient top eigenvalue computation for Hessian of neural networks

I train a neural network - one of the Resnet variations ($\approx 10^7$ parameters) on the CIFAR-10 dataset - and after each epoch, I would like to find the smallest/largest eigenvalues of its Hessian....
2
votes
0answers
24 views

Simple back propagation example

Sorry if this is too simplistic of a question, but over the last couple of months I have been working through the course mathematical foundations of machine learning at my college. I think I am really ...
2
votes
0answers
34 views

How do convolution layers work?

So let's say I have a 4x4 image with 3 channels. So we have 3 4x4 matrices where each matrix represents a channel. Now let's say I also have a 3x3 kernel. I know that for convolution layers, I have to ...
2
votes
1answer
109 views

Learning the activation function in a neural network?

Neural networks use specific activation functions, commonly used ones are tanh, ReLu. I have seen that people have experimented with continuously parametrices activation functions, for instance here. ...
2
votes
0answers
34 views

Multilayer Perceptrons for solving variational problems

Can we use a multilayer perceptron to solve variational problems? By variational problem I mean something we might encounter in the calculus of variations, for example the geodesic problem: given two ...
2
votes
0answers
111 views

Approximate dot product between neural network output layer's parameter vector and input activations with winner-take-all hashing

In the paper Deep Networks with Large Output Spaces, Vijayanarasimhan et al. describe their approach to approximating the dot product between a neural network's output layer's parameter vector and ...
2
votes
0answers
27 views

How often should I read out information from an echo state recurrent neural network?

Recurrent neural networks makes it possible to implement some kind of memory, which can be very useful for a lot of tasks, incl. (but not limited to) robot control, which I am interested in. For ...
2
votes
0answers
247 views

Multilayer perceptron memory requirements

How much memory do we need to train a multilayer perceptron? I've started to figure this out myself, but I'm stuck. I have one-layer MLP. Each training example is a vector of 100 real numbers in ...
2
votes
0answers
115 views

Trying to understand Le's “cat paper”

QV Le et al. show in Building high-level features using large scale unsupervised learning (2012) how to use unsupervised learning of a deep neural net to recognize faces and cats. Working through this ...
2
votes
0answers
35 views

is mappings optimization problems onto neural networks still a thing?

Is mappings optimization problems onto neural networks still a thing? I only found papers are written prior to 1999. These old papers mostly deal with Hopfield network, which i read is obsolete and ...
2
votes
0answers
494 views

Class activation mapping using inception-v3 network

I am using the technique of this paper: https://arxiv.org/pdf/1604.07953.pdf to localize food objects. The difference is that I am using the next generation of google-net, inception-v3 network. For ...
2
votes
0answers
66 views

How to best model multidimensional, continuous, non-convex “shape” as neural network?

I have: set of n-dimensional points that I know are inside of the shape n >= 18, range on all dimensions has upper bound and lower bound (no axis goes to infinity). shape is pretty large in this n-...
2
votes
0answers
91 views

KDD Machine Learning using K-NN Algorithm Classification Problem

I'm trying to solve a classification problem from the KDD cup archive of 2004. Details can be found here: KDD 2004 Archive I'm only dong the particle physics part. The description of dataset is as ...
2
votes
0answers
12 views

What would happen if a multi-layer perceptron is insufficient to learn a function?

Suppose that something is wrong with its configuration (ie topology), for example there are too few neurons in a certain layer, or not enough layers. I have an intuition that some neurons' "delta" ($\...
2
votes
0answers
481 views

How to select the maximum weight value for a bias node in a neural network?

I'm programming a neural network. I know that I should initialize the network by picking random weights. How do I pick a random weight for the connections to bias nodes? What distribution should I ...
2
votes
0answers
105 views

How can you use HMMs and ANNs for on-line handwriting recognition?

On-line handwriting recognition is the task of converting a series of $(x(t),y(t))$ coordinates to symbols and words. In contrast to off-line handwriting recognition, where you only have a bitmap of ...
1
vote
1answer
13 views

How many pixels support each neuron in multi-layer CNN?

I'm studying for a computer vision module and I'm on the deep learning topic, in one past paper we have the following question: Given that a convolutional neural network has five convolution layers (...
1
vote
0answers
23 views

Discrete action space for MADDPG

I am trying to apply MADDPG, a policy gradient algorithm that uses centralized training and decentralized execution, to a project. In the work of Lowe et al., the actor returns a pmf for a discrete ...
1
vote
0answers
89 views

Which features can be considered for neural network based SAT solving?

I'm trying to implement SAT solver, based on backtracking algorithm and BCP. This SAT solver is trying to pick one literal from each clause, from 3-CNF SAT instances. I've implemented a neural network ...
1
vote
0answers
39 views

In a machine learning system, why use differentially private SGD if our input data is already perturbed by a DP mechanism?

I'm trying to implement my own version of a deep neural network with differential privacy to preserve the privacy of the parties involved in the training dataset. I'm using the method by Abadi et al. ...
1
vote
0answers
34 views

How can Kneser-Ney Smoothing be integrated into a neural language model?

I found a paper titled Multimodal representation: Kneser-Ney Smoothing/Skip-Gram based neural language model. I am curious about how the Kneser-Ney Smoothing technique can be integrated into a feed-...
1
vote
0answers
20 views

Why do we need to change the (weight decay) regularization parameter when changing the number of inputs that neural network is being trained with?

I am currently working my way through Michael Nielsen's ebook Neural Networks and Deep Learning and I am reading about overfitting and (L2) regularization. In this subsection, the process of L2 (a.k.a ...
1
vote
0answers
8 views

Is there a way to connect a deep language model output to input?

In models like GPT-2, TXL and Grover, is there a good way to know which input weights (tokens) resulted in each token of the output?
1
vote
0answers
23 views

Value flow (and economics) in stacked reinforcement learning systems: agent as reinforcement environment for other agents?

There is evolving notion of stacked reinforcement learning systems, e.g. https://www.ijcai.org/proceedings/2018/0103.pdf - where one RL systems executes actions of the second RL system and it itself ...
1
vote
1answer
189 views

What does “Temporal extent” mean?

I am reading Long-term Temporal Convolutions for Action Recognition and under the Section 3.1, I read this: To investigate the impact of long-term temporal convolutions, we here study network inputs ...
1
vote
0answers
150 views

Converting (reverse-engineering) Turing machine into program or most concise algorithm?

It is known that every program or every algorithm can be converted to Turing machine. But what about the reverse process? Is there algorithm (or research trend that considers such algorithm) to ...
1
vote
0answers
452 views

Why is the method of im2col with GEMM is more efficient than the method of direction implementation with SIMD in CNN

The convolutional layers are most computationally intense parts of Convolutional neural networks (CNNs).Currently the common approach to impement convolutional layers is to expand the image into a ...
1
vote
0answers
153 views

Proof of perceptron convergence theorem for ZERO threshold?

The generalized perceptron convergence theorem is for a defined threshold T. When you do the maths it all comes to an upper bound and a lower bound. The lower bound looks like this! Therefore ...
1
vote
0answers
387 views

How can node2vec help find similar “roles” within a graph (nodes whose connections have similar structure within the graph)?

I have a question on the node2vec algorithm described in this paper. Node2vec is a deep learning algorithm that word2vec to graphs to learn embeddings. The authors claim that it can help find nodes ...
1
vote
0answers
33 views

How would you go about creating a algorithm that should generate a shakespearean sonnet on any given theme

I need to create an algorithm that is going to create a shakespearean sonnet for a specific theme. This theme should be generated out of twitter tweets that have some hashtag. My current idea goes ...
1
vote
0answers
712 views

Backward mapping with bilinear sampler

I have some experiences with Convolutional Neural Networks before. I have a question regarding the Bilinear Sampler used in "Unsupervised Monocular Depth Estimation With Left-Right Consistency" (the ...
1
vote
0answers
42 views

Does a deep feedforward neural net approximate a (single or multivariate)polynomial?

Does composition of several linear transformation plus non-linear activation function in each layer and different layers (as they are in feedforward neural net) represent or approximate a ...
1
vote
0answers
167 views

Recurrent neural networks (Hopfield-like) with short limit cycles

Standard Hopfield networks exhibit stable patterns (states) which are attractors of a dynamic system. I wonder how to modify standard Hopfield networks such that they exhibit stable limit cycles as ...
1
vote
0answers
22 views

How do RNN's map variable-length sequences to variable-length sequences?

According to Karpathy's blog "The Unreasonable Effectiveness of Recurrent Neural Networks", recurrent neural networks can map variable-length sequences to variable-length sequences, as shown by the ...
1
vote
0answers
202 views

Distributing Neural Network Training World-Wide Using Blockchain?

I just finished reading a white-paper from a recent AI startup. The company, Deep Brain Chain, wants to distribute neural network training over computers worldwide, using blockchain technology. Here ...
1
vote
0answers
137 views

adjusted fitness in NEAT algorithm

I'm learning about NEAT from the following paper: http://nn.cs.utexas.edu/downloads/papers/stanley.ec02.pdf I'm having trouble understanding how adjusted fitness penalizes large species and prevents ...
1
vote
0answers
51 views

Detecting shapes using CNNs in respect to other shapes in the image

I'm solving a problem of detecting an optic disc in a retina image. As you can see from the image: the optic disc is the epicentrum of the blood vessels, has an irregular circular shape and has a ...