Questions tagged [neural-networks]
Network structure inspired by simplified models of biological neurons (brain cells). Neural networks are trained to "learn" by supervised and unsupervised techniques, and can be used to solve optimization problems, approximation problems, classify patterns, and combinations thereof.
492
questions
0
votes
0
answers
10
views
Dilated LSTM vs LSTM
I'm currently reading a paper for a literature review related to NLP and its research employs a dilated LSTM model.
What is the difference between this and a regular LSTM, or more specifically what ...
- 11
1
vote
0
answers
28
views
Understanding gradient flow of a linearized wide neural network
I've been trying to fully understand the paper "Wide Neural Networks of Any Depth Evolve as
Linear Models Under Gradient Descent" (available here), but I'm stuck on the linearization part, ...
- 11
1
vote
0
answers
34
views
At what precision are GPT's parameters stored?
OpenAI's technical paper on GPT-3 says that GPT-3 has 175 billion parameters. Several sites (here, here) claim that these parameters are stored as single-precision floating-point numbers and so ...
- 1,034
0
votes
0
answers
10
views
Neural Net Back-propagation: Question on Wikipedia math
I'm reading the Wikipedia article on Backpropagation and have some questions on the math.
$$
\frac{d C}{d a^L}\cdot \frac{d a^L}{d z^L} \cdot \frac{d z^L}{d a^{L-1}} \cdot \frac{d a^{L-1}}{d z^{L-1}}\...
- 175
1
vote
1
answer
32
views
meaning behind activation function
I have been recently thinking about activation functions and the explainability.
For sigmoid and tanh activation functions, I am thinking of them to be similar to logistic regression as the output of ...
- 11
0
votes
0
answers
38
views
Create a simple Neural Network of n layers in python from scratch with numpy to solve XOR example problem using Batch Gradient Descent
I'm a young programmer that was interested by machine learning. I watched videos and read articles about the theory behind simple neural networks. However, I can't manage to set it up correctly. I've ...
- 1
0
votes
1
answer
20
views
Calculation in paper of learning rate of neural network
The rate at which a neural network learns something depends on the loss and the backpropagation functions,neural network with different loss and backpropagation functions learn differently at a ...
- 97
2
votes
1
answer
267
views
As far as we know, what does GPT-4's training data look like?
I was asking ChatGPT local history questions, because I knew many of the answers and could test the robot. But lots of details were wrong and while it can write citations, it won't disclose its ...
- 121
0
votes
0
answers
16
views
Is proof based calculus more useful to become an AI researcher and an advanced specialist?
I don't know which single variable calculus course to choose: mit 18.01 or mit 18.014( proof-based calculus). I want to become an advanced specialist
0
votes
0
answers
16
views
Bellmann Error with Time Step
I define the return $G$ as the discrete time integral of the reward $G = \Delta t \sum_{t = 0, ..., T}r_t$. In supposing $\Delta t = 1$ throughout, my instructor wrote the following formula for the ...
1
vote
0
answers
37
views
Draw the decision boundary of an neural network
I'm really confused when it comes to what the difference between bias and threshold is. I have read that they are basically the same thing and that they serve the same purpose. Being on opposite sides ...
- 21
2
votes
0
answers
17
views
Finding Large Subset of Variables Approximately Equal, Low-Cost with Neural Network
I was reading David J.C. MacKay's Information Theory, Inference and Learning Algorithms(pdf available in the link), and in section 16.4 of Chapter 16: Message Passing, he writes about how only ...
- 21
1
vote
0
answers
136
views
Tabular Meta-Learning in RL
There are various meta-learning algorithms in RL that are proposed for settings when we have a (deep) neural network and the policy (or the value function) are parameterized as such.
Can these methods ...
- 175
0
votes
1
answer
33
views
How sparsity term in loss function for sparse autoencoder is making hidden units inactive?
I am working on a Sparse Autoencoder but Andrew NG's notes are hard to understand.
My question is about the following equation: Loss Function.
In sparse autoencoder, the goal is to inactive some ...
2
votes
3
answers
166
views
Does ChatGPT use specific sub-programs
I have had a somewhat hard time trying to understand how ChatGPT can "solve" some tasks that cannot be entirely cast as language-model-based rephrasing of textual subsets of the internet ...
- 177
0
votes
1
answer
22
views
Equivalence of echo state networks and DFAs/NFAs
Echo state networks are theoretically equivalent to DFAs/NFAs, but how would you use an ESN to parse a regular language? Would you just feed many different input strings, some from the language and ...
- 11
0
votes
1
answer
3k
views
What are the 175 billion parameters used in the GPT-3 language model?
I am currently working my way through Language Models are Few-Shot Learners
, the initial 75-page paper about GPT-3, the language learning model spawning off into ChatGTP.
In it, they mention several ...
- 2,163
-1
votes
1
answer
33
views
Convolutional neural network for non-image data
I am working on a project for the prediction of preeclampsia, and one of the algorithms to be used is the convolutional neural network. I am a beginner in machine learning, and I am struggling to ...
0
votes
0
answers
16
views
Computer Vision techniques to find the slope of the buildings with respect to the ground?
I want to find the slope of the building with respect to the ground which means that I need to find how tilted is the building with respect to the ground through computer vision or any other ...
1
vote
1
answer
36
views
Are there already Neural Network AIs that can play object-finding games?
Neural networks are good at image analysis. They are also being used in video games which could outperform professional human players. I wonder if they can also play a purely object-finding game where ...
0
votes
0
answers
19
views
Link prediction in a network using GNNs versus Matrix Completion and other methods
I am looking at some social network analysis and I need to do link prediction. By link prediction I mean that I only have a subset of the relationships in the network, and I need to make predictions ...
- 151
2
votes
1
answer
379
views
What are common ways to distinguish between local minima and long run-time in hyperparamteer optimization
I'm using Bayesian optimization for a pretty cost expensive function in the context of neural networks in order to optimize hyperparameters for the neural net.
Is there a general, quantitative way to ...
1
vote
1
answer
144
views
does local search's State-Flipping Algorithm always terminate?
I am looking at the state flipping algorithm, an algorithm that tries to find a stable configuration in a Hopfield network.
The algorithm simply flips the state of unsatisfied nodes as long as the the ...
- 310
1
vote
1
answer
23
views
Convolutional Neural Network : how to ignore swaths of uninteresting areas? Or make it focus to interesting regions only?
Imagine you trained a text-from-image reader, similarly to the one described in this article https://medium.com/analytics-vidhya/image-text-recognition-738a368368f5
Now you want to port the trained ...
- 11
1
vote
1
answer
34
views
Why does Calibration of NN not lead to higher accuracy?
I have a question about the use of calibration in neural networks and its relation to accuracy. In Guo et al. it says: "Because the parameter T does not change the maximum of the softmax function,...
- 11
1
vote
0
answers
28
views
Long and short memory in reinforcement learning Connect 4 AI
I'm writing an AI based on reinforcement learning to play Connect 4. That's my second bot and attempt to RNN and AI (first was copy a code of snake RNN AI from youtube) and I'm looking for some ...
- 11
1
vote
1
answer
39
views
Which artificial neural network model can I use when trying to predict gene expression?
I need help choosing a model for my data. I have 50 genes that I need to test for their expression levels. I was thinking of implementing either an SVM or a multilayer perceptron model where the ...
- 13
1
vote
1
answer
112
views
Is Self-Modifying Turing Machine equivalent to NTM or TM?
Let SMTM be Turing Machine, but the commands recorded in which can change to others in some random way (for example, choose with a 50/50 probability the command to move to the right or move to the ...
- 113
0
votes
0
answers
12
views
Reusability of neural network architecture
I have created a neural network architecture by considering the data for one study area. Have performed grid-search to tune almost all the hyperparameters of the model. I want to use the same network ...
- 1
1
vote
1
answer
36
views
Function or signal as machine learning model output
I am new to machine learning. I've seen that machine learning models such as ANN or SVM can be used to predict scalar outputs taking scalar inputs.
I am trying to create a model that has a number of ...
- 113
1
vote
0
answers
77
views
RMSProp Momentum and Decay
I'm making an application of MobileNetV2 and according to their article:
We train our models using
TensorFlow. We use the standard RMSPropOptimizer with both decay and momentum set to 0.9.
We use ...
- 11
0
votes
1
answer
37
views
How can I modify this detail in the article ""Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications"?"
This question is about de paper Extending Multi-Sense Word Embedding to Phrases and Sentences
for Unsupervised Semantic Applications.
I am interested in the transformer part of the paper and the main ...
- 137
1
vote
2
answers
44
views
Are the weights between hidden layers changed in backwards propogation? If so, how
If so, I don't understand how, as a large part of back propogation is knowing what the 'real' answer is in comparison to 'predicted' answer. With 1 neuron in a hidden layer, we do not know the 'real ...
1
vote
1
answer
110
views
Predicting continuous variables using image and text input to neural network
I am looking to create a neural network architecture that takes two kinds of inputs: image and text, and outputs a predicted continuous variable. More specifically, I will have multiple images of a ...
- 11
1
vote
0
answers
142
views
How should I create a word-embedding an NLP model recognizing HTML elements?
I'm currently doing a small-time project where I have to deliver a model which can classify specific elements on a web-page using the HTML code.
For this, I have considered using the HTML tags for ...
0
votes
3
answers
151
views
Mutual Friends in a Network?
I always seem to have trouble finding a formal way to analyze this (be through proofs or whatever).
The problem statement is as such:
If A and B are friends, and B and C are friends, then A and C are ...
4
votes
1
answer
39
views
What is the primary reference for the observation/discussion of how neural networks struggle with ambiguous training datasets?
It is known that neural networks, such as convolutional neural networks, struggle with pattern recognition if training sets contain ambiguities (i.e. several labels can correspond to one and the same ...
1
vote
0
answers
11
views
How long can the short memory last in the RNN?
For a recurrent neural network, the LSTM was a model of how the network worked. However, consider the case where an input was a long paragraph or even an article.
$$c_1c_2...c_n$$
where $c_i$ were ...
1
vote
1
answer
34
views
Convergence of graph neural networks (GNNs)
I'm new to the area, and we don't have a course on graph neural networks at our university. However, I will still like to know the main theoretical results when considering convergence of graph neural ...
- 11.5k
1
vote
2
answers
39
views
Can we supervise on the hidden states of RNN?
I'm trying to generate some history-dependent model with machine learning, whose underline physical model has a clear definition of its "internal state variable" (a state derived from ...
- 13
2
votes
1
answer
287
views
Is ANN a data structure or an algorithm?
When I read about Artificial Neural Networks (ANN), no one says what ANN is.
For instance, Wikipedia says:
Artificial neural networks (ANNs), usually simply called neural networks (NNs), are ...
- 153
1
vote
1
answer
239
views
How many pixels support each neuron in multi-layer CNN?
I'm studying for a computer vision module and I'm on the deep learning topic, in one past paper we have the following question:
Given that a convolutional neural network has five convolution layers (...
- 11
4
votes
0
answers
62
views
Universal approximation bounds of the form $\|f(x)-\hat{f}(x;w)\|\leq \varepsilon \|f(x)\|$
It is known that for every $\varepsilon>0$ there is an appropriate neural network architecture, such that one can approximate any continuous function $f:[0,1]^n\to[0,1]^m$ by the neural network ...
- 225
2
votes
0
answers
152
views
What are open-loop and closed-loop modes of neural networks?
I came across the following line in the book ‘Deep Learning (Ian Goodfellow) 10.2.1, pg 374;
The disadvantage of strict teacher forcing arises if the network is going to be later used in an open-loop ...
- 141
2
votes
1
answer
344
views
How to transform an Abstract Syntax Tree (AST) to an Abstract Binding Tree (ABT)? (for machine learning fo theorem proving)
I was reading the HOList paper that applies Graph Neural Networks (GNNs) to the HOL Light (HOList) data set and benchmark for ML for theorem proving. They describe their results etc but there is no ...
- 2,960
2
votes
0
answers
150
views
Discrete action space for MADDPG
I am trying to apply MADDPG, a policy gradient algorithm that uses centralized training and decentralized execution, to a project. In the work of Lowe et al., the actor returns a pmf for a discrete ...
2
votes
1
answer
72
views
Raising to the T in machine learning
What does it mean when in a machine learning paper there is
$(arg)^{T}$, what does the T does to an arg in this 3b1b video on neural networks he puts the: $(w^{l-1})^{T}$
- 21
1
vote
0
answers
144
views
Which features can be considered for neural network based SAT solving?
I'm trying to implement SAT solver, based on backtracking algorithm and BCP. This SAT solver is trying to pick one literal from each clause, from 3-CNF SAT instances. I've implemented a neural network ...
- 71
0
votes
1
answer
48
views
Linear Discriminant Method as Pre-processing for inputs going into a Neural Network?
As the title suggests, I am curious whether Linear Discriminant Method can be performed on a dataset as pre-processing before putting the reduced-dimensional data as input for a neural network for ...
- 5
1
vote
1
answer
25
views
How to give the nearest text that the user has written
In my database I have 1500 food dishes, not all are single words, there are compound words like "cheese with dried fruit and nuts". And I have them in 5 languages (de, en, fr, es, it).
My ...
- 111