People who code: we want your input. Take the Survey

# Questions tagged [gradient-descent]

The tag has no usage guidance.

38 questions
Filter by
Sorted by
Tagged with
198 views

### Algorithms for curve construction

I am interested in algorithms that construct continuous curves between two points in such a way that minimizes an energy functional of the curve. What sort of algorithms are most used for such tasks? ...
2k views

### How to show that cross entropy is minimized?

This Question is taken from the book Neural Networks and DeepLearning by Michael Nielsen The Question: In a single-neuron ,It is argued that the cross-entropy is small if σ(z)≈y for all training ...
1k views

### Why update weights and biases after training a Neural Network on whole set of training samples

I am reading the book Neural Networks and Deep Learning by Micheal Nielsen. In the second chapter of his book, he describes the following algorithm for updating weights and biases for a neural ...
450 views

751 views

### Why do we use the log in gradient-based reinforcement algorithms?

I've been reading some papers on reinforcement learning. $$\Delta w=\frac{\partial ln\ p_w}{\partial w}r$$ I often see expressions, similar to the above one, where the weights (denoted by $w$) are ...
5k views

### How to optimize a function by maximizing 1 variable and minimizing another?

Problem I want to implement an optimization algorithm for my file transfer program. The program buffers data in a local file before uploading to central server periodically and it compresses the ...
89 views

### Is there a universal learning rate for NeuralNetworks?

I'm currently creating a NeuralNetwork with backpropagation/gradient descent. There is this hyperparameter introduced called "learning rate" (η). Which has to be chosen to guarantee not overshooting ...
798 views

### Calculating gradient in a neural net using batches

I am a CS student learning about neural nets. Currently I am confused about how to train a neural net in batches. If I calculate error in a batch, I will get a vector of errors e.g. real1 - predicted1,...
63 views

### How is momentum an approximation of Hessian based optimization?

In the answer to "what is the Hessian" at this site: https://stackoverflow.com/questions/23297090/how-calculating-hessian-works-for-neural-network-learning the person answering the question ...
27 views

### Inverse kinematics step

I am working on an implementation of inverse kinematics using the jacobian transpose method. The implementation seems to be working as it does find the "theta" vector, although sometimes it might take ...
51 views

39 views

### In a machine learning system, why use differentially private SGD if our input data is already perturbed by a DP mechanism?

I'm trying to implement my own version of a deep neural network with differential privacy to preserve the privacy of the parties involved in the training dataset. I'm using the method by Abadi et al. ...
37 views

7 views

### Prophet Model in Batch Implementation

I am newby at Prophet model. I was wondering Prophet uses stochastic or batch approach while in training. If it uses stochastic that means there is nothing to do convert and compute mini batch ...
41 views

### Gradient Descent in MLPs using Computational Graphs

I'm working through Deep Learning by Goodfellow et al. The textbook introduces backpropagation for MLPs in page 203 (http://www.deeplearningbook.org/contents/mlp.html). However, it does not expand ...
69 views

### Gradient Descent With Constraints

I'm playing around with some historical stock data and attempting to optimize a portfolio. I essentially have created a function that generates certain statistics about a portfolio (right now it's ...
386 views

### What is role of parameter learning rate, lr, and momentum constant, mc in Neural Networks?

can anyone describes the more simplified mathematical formulation of learning rate, lr, and momentum constant, mc in Neural Networks while training the data?
1k views

### Classifying responses into yes/no

So my problem is as follows: I get responses (such as "yeah whatever", "yes do it", "no don't do it", "nah", "yeah do it" etc.) and I need to classify them into either "yes" or "no" i.e. a binary ...