Questions tagged [neural-networks]

Network structure inspired by simplified models of biological neurons (brain cells). Neural networks are trained to "learn" by supervised and unsupervised techniques, and can be used to solve optimization problems, approximation problems, classify patterns, and combinations thereof.

Filter by
Sorted by
Tagged with
2
votes
1answer
34 views

object image segmentation

I have 2 different datasets with similar objects, one where each object is 50 pixels wide and the other where they are 150 pixels. Each photo is 512x512 for both datasets. These two datasets have the ...
0
votes
1answer
34 views

What Happens if I swap the forget gate and update gate in LSTM model?

Consider the following eqautions used in LSTM ( taken from Andrew ng's course on Sequential model) In an LSTM model, LSTM Cell has three inputs at any time step t Input($X_t , a^{(t-1)}, C^{(t-1)})$, ...
1
vote
1answer
27 views

How can one measure the time dependency of an RNN?

Most of the discussion about RNN and LSTM alludes to the varying ability of different RNNs to capture "long term dependency". However, most demonstrations use generated text to show the ...
1
vote
1answer
38 views

Why isn't there just one “keystone” activation function in Neural Networks?

This article says the following: Deciding between the sigmoid or tanh will depend on your requirement of gradient strength. I have seen (so far in my learning) 7 activation functions/curves. Each ...
0
votes
2answers
53 views

Why don't they use all kinds of non-linear functions in Neural Network Activation Functions? [duplicate]

Pardon my ignorance, but after just learning about Sigmoid and Tanh activation functions (and a few others), I am wondering why they choose functions that always go up and to the right? Why not use ...
1
vote
1answer
49 views

How was the σ function chosen to extend the perceptron?

I am just reading about perceptrons in more depth, and now onto Sigmoid Neurons. Some quotes: A small change in the weights or bias of any single perceptron in the network can sometimes cause the ...
1
vote
0answers
39 views

In a machine learning system, why use differentially private SGD if our input data is already perturbed by a DP mechanism?

I'm trying to implement my own version of a deep neural network with differential privacy to preserve the privacy of the parties involved in the training dataset. I'm using the method by Abadi et al. ...
0
votes
0answers
37 views

Using the features embedding of the output from a transformers to represent probabilities of categorical data

I was considering using a transformer, on input data which can be represented as an embedding, so I can use the attention mechanism in the transformer architecture. As my data is of variable input and ...
1
vote
0answers
34 views

How can Kneser-Ney Smoothing be integrated into a neural language model?

I found a paper titled Multimodal representation: Kneser-Ney Smoothing/Skip-Gram based neural language model. I am curious about how the Kneser-Ney Smoothing technique can be integrated into a feed-...
1
vote
2answers
58 views

Why doesn't relu cause back propegation to get stuck

Say you have a neural net that is being trained using back propagation and you are using relu activation. Say the input to a node is a weighted sum of the previous layer with a bias term and say for a ...
4
votes
1answer
24 views

Are non-ordered (Slabbed) Neural Networks in wide modern use and what does Fiesler (1994) mean by Clamping Function and Ontogenic function?

In Fiesler (1994) Neural Network Classification and Formalization, he talks a lot about a more general version of neural networks, one that is not ordered into layers, but rather the network is called ...
0
votes
0answers
51 views

Delta rule for binary step function

I have a question of understanding about the Delta Rule: $\Delta w_i = (y - \hat{y}) \times x_i$ Why does $x$ have to be multiplied again after the difference? If the input is $0$, the product of $w$...
2
votes
1answer
62 views

Can you apply neural networks to design algorithms?

I’m kind of a newbie to neural networks (and CS in general) but I was wondering if there are any methods to apply them in such a way with the aim of producing algorithms that solve difficult math ...
2
votes
2answers
61 views

Zero-Sum Games and Halting Problem

Wikipedia states on the page of the halting problem, "For any program f that might determine if programs halt, a "pathological" program g called with an input can pass its own source and its input to ...
0
votes
1answer
53 views

Performing semantic segmentation on images on word documents

At the moment, I'm making a comparative analysis between two different neural networks for a class. The problem they are both trying to solve is detecting tables on documents. Below is an example of ...
5
votes
5answers
327 views

fast and stable x * tanh(log1pexp(x)) computation

$$f(x) = x \tanh(\log(1 + e^x))$$ The function (mish activation) can be easily implemented using a stable log1pexp without any significant loss of precision. Unfortunately, this is computationally ...
0
votes
0answers
42 views

How to find clusters of a set of points in n-dimensional space that are furthest from unwanted points

I have a list of 25 points and their coordinates in a 512-dimensional space. I have 8 target points and 17 points I need to avoid (the 17 points to avoid also have differences in priority of how ...
2
votes
1answer
86 views

What are some complexity classes for neural networks?

Turing machines and neural networks are equivalent in their expressive powers, but as models of computation they are different. Turing machines come pre-configured with their transition functions ...
1
vote
0answers
20 views

Why do we need to change the (weight decay) regularization parameter when changing the number of inputs that neural network is being trained with?

I am currently working my way through Michael Nielsen's ebook Neural Networks and Deep Learning and I am reading about overfitting and (L2) regularization. In this subsection, the process of L2 (a.k.a ...
2
votes
0answers
24 views

Simple back propagation example

Sorry if this is too simplistic of a question, but over the last couple of months I have been working through the course mathematical foundations of machine learning at my college. I think I am really ...
1
vote
1answer
26 views

Learning a specific functional form with machine learning

Suppose I have only three independent features (x, y, z) as the input to some machine learning routine (e.g. neural network). From some domain knowledge, I know ...
0
votes
1answer
11 views

Intersection of two generative neural networks

I have an input x (a word), and I have a neural network Y such that Y(x), which is an image always satisfies F(x), and another neural network Z such that Z(x), which is also an image, always satisfies ...
0
votes
0answers
26 views

Artificial Neuron: question on definition and inhibited signals

This is an artificial neuron of a NOT function from Rojas's ML book. I have a question on it's behavior. It's my understanding that the neuron produces a signal if it's inputs $x_1$ summed are $\geq ...
0
votes
1answer
73 views

Mcculloch and Pitts Neurons: Is there errata in this article?

The paper A LOGICAL CALCULUS OF THE IDEAS IMMANENT IN NERVOUS ACTIVITY has the following description of neurons, which I question if there is errata or perhaps I need more information to understand ...
1
vote
1answer
81 views

Approximating Deep Neural Networks (DNNs) with Binarized Neural Networks (BNNs)

I am working currently as a research intern on Binarized Neural Networks where the weights and the activations of the network are binary. The architecture of this type of networks makes them memory ...
1
vote
2answers
86 views

Can neural network process randomness?

So the question is : Is it theoretically possible to feed a neural network with some random values to expect an output since randomness is a lack of knowledge in most case. For this question, I've ...
2
votes
2answers
104 views

Being stuck and frustrated with my masters project

I'm doing a masters in CS that requires me to implement from scratch most of the neural network models and because python libraries aren't applicable to what i want. The problem is that i don't feel ...
2
votes
1answer
67 views

Energy consumption off CNN models

I want to calculate/estimate the energy consumption for the different convolutional neural networks. Is there any possibility to measure the energy consumed by AlexNet for example with a tool or with ...
0
votes
1answer
58 views

Understanding how convolution of images work in cnn

I'd like to understand convolutional neural network. Consider this quote from Stanford CS notes on Convolutional Neural Networks for Visual Recognition (CNNs/ConvNets): Example Architecture: ...
0
votes
1answer
25 views

Using a point as an input for a perceptron

Can a point be used as the input for a perceptron/neural net? The relationship between the two numbers that make up a 2D point does not affect the output, but does this not matter when the ...
1
vote
1answer
71 views

How does the forget layer of an LSTM work?

Can someone explain the mathematical intuition behind the forget layer of an LSTM? So as far as I understand it, the cell state is essentially long term memory embedding (correct me if I'm wrong), ...
1
vote
1answer
43 views

Text detection in computer vision

I'm curious about the way text recognition works in machine learning(or more generally, the question of object vs not object) in computer vision. How are systems trained when the not-object data set ...
1
vote
1answer
91 views

Weighted Average of Neural Networks with Cross Entropy Cost Function

What is the best way to create an ensemble of neural networks utilising weighted averaging when these networks were trained to minimise cross entropy error function? The literature I found (e.g. [1]) ...
2
votes
1answer
79 views

Weighted Average of Multi-Output Neural Networks

[1] discusses how to construct an ensemble of neural networks by giving each network a certain weight $\alpha_i$: \begin{equation} f_\mathrm{GEM}(\boldsymbol{x}) = \sum_{i=1}^N \alpha_i f_i(\...
2
votes
1answer
36 views

Why do we need to take the derivative of the activation function in backwards propagation?

I was reading this article here: https://towardsdatascience.com/how-does-back-propagation-in-artificial-neural-networks-work-c7cad873ea7. When he gets to the part where he calculates the loss at ...
0
votes
0answers
70 views

Are neural network latent representations fitting a Gaussian distribution?

Neural network latent (or pre-activation) representations are the weighted sums of inputs to neurons in hidden layers before applying an activation function. The vector of representations of neurons ...
0
votes
0answers
39 views

Cross entropy minimization

I just read about Cross-entropy cost function for my work. and I see a notation that said "the minimization of cross entropy of some data is equal to maximization of their log likelihood." How can I ...
0
votes
1answer
53 views

How does image reconstruction take place in neural network?

I am reading through and thinking about how neural network works and have been reading about convolutional neural networks (CNN). I am particularly interested in image filtering (or enhancing) using ...
0
votes
1answer
84 views

Explain this Neural Network used for Knowledge Graph Embedding

I am interested in better understanding the neural network used in the paper LogicENN A Neural Based Knowledge Graphs Embedding Model with Logical Rules. To my knowledge this is the most advanced and ...
1
vote
0answers
8 views

Is there a way to connect a deep language model output to input?

In models like GPT-2, TXL and Grover, is there a good way to know which input weights (tokens) resulted in each token of the output?
1
vote
1answer
52 views

Neural network for PDE: Should we train the PDE using more initial and boundary data at the beginning?

I was trying to solve a partial differential equation (PDE) using a neural network. The solution to the PDE is not unique unless the boundary condition is determined. In my case, the neural network ...
0
votes
1answer
46 views

Which algorithm for predicting the next word(s) based on previous words, given a sentence?

I want to input some words, and out comes the next word(s). Neural nets are really hot at the moment, and I'm afraid of throwing a neural net at something, when one is not really needed. Or... maybe ...
2
votes
2answers
33 views

Neural network game players and incremental updates

Neural networks in recent years have been successfully used for gameplaying. A difference between games and e.g. image processing is that the game boards get updated incrementally. Do any neural ...
0
votes
0answers
37 views

How to transform an arbitrary graph into a fixed vector representation?

Actuality I work in computer vision, specifically on a problem known as "scene graph modeling." This problem aims to convert an image $I$ in a graph $G=(V,E)$ where the nodes $V$ represent the objects ...
0
votes
1answer
28 views

How to initialize the first h in an RNN?

Take a Vanilla RNN represented by the function $h_t = f(h_{t-1}, x_t)$, how do you determine $h_0$? Edit: This answer over on the stats page has helped.
1
vote
0answers
23 views

Value flow (and economics) in stacked reinforcement learning systems: agent as reinforcement environment for other agents?

There is evolving notion of stacked reinforcement learning systems, e.g. https://www.ijcai.org/proceedings/2018/0103.pdf - where one RL systems executes actions of the second RL system and it itself ...
1
vote
0answers
63 views

Solving analytic gradient of loss function for neural networks [closed]

Please note that I am talking in about theory rather than ''what someone would do in a real, practical situation''. Given a multi-layer Perceptron with at least 1 hidden layer, and sigmoid (or other ...
1
vote
3answers
3k views

Guided Backpropagation in Deep Neural Networks

I am reading about a method called guided backpropagation. https://ramprs.github.io/2017/01/21/Grad-CAM-Making-Off-the-Shelf-Deep-Models-Transparent-through-Visual-Explanations.html#deconv-and-guided-...
1
vote
1answer
49 views

What is the most efficient way to test whether a set $X \subset \{0, 1\}^n$ and its complement $\{0, 1\}^n \setminus X$ are linearly separable?

I am interested in algorithms that have optimal running time, and ideally which are also very easy to implement. If you can also give some tips on how to implement the algorithm(s) you mention in the ...
0
votes
0answers
168 views

Sorting using AI / neural net

I have a search operation taking place on a server that essentially queries images using OpenCV against other images from a database. Since each image query ...

1
2
3 4 5
10