Questions tagged [gradient-descent]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
1 answer
135 views

What does RSGD stand for?

I'm reading a paper that involves an algorithm for RSGD. It's clearly a form of stochastic gradient descent, but I can't find what the R stands for. The authors provide their own implementation of it, ...
Dalop's user avatar
  • 125
1 vote
0 answers
31 views

Understanding gradient flow of a linearized wide neural network

I've been trying to fully understand the paper "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent" (available here), but I'm stuck on the linearization part, ...
user161590's user avatar
0 votes
0 answers
66 views

Create a simple Neural Network of n layers in python from scratch with numpy to solve XOR example problem using Batch Gradient Descent

I'm a young programmer that was interested by machine learning. I watched videos and read articles about the theory behind simple neural networks. However, I can't manage to set it up correctly. I've ...
NolanGio's user avatar
0 votes
0 answers
55 views

How to calculate the upper bound of the gredient of a multi layer ReLu neural network

Layers: We shall denote in the following the layer number by the upper script $\ell$. We have $\ell=0$ for the input layer, $\ell=1$ for the first hidden layer, and $\ell=L$ for the output layer. The ...
river7816's user avatar
2 votes
0 answers
21 views

Convergence rate of quasi-newton method for non-convex objective function

Consider a real-valued $L$-smooth and non-convex objective function $f: \mathbb{R}^n \mapsto \mathbb{R}$. There exists a bound on number of iterations in order to find a (local) minima using ordinary ...
Leeseok Kim's user avatar
1 vote
1 answer
105 views

Why when a function is quadratic, the approximation by Newton's method is exact, and the algorithm converges to the global minimum in a single step?

Suppose we want to find the value of $x$ that minimizes $$ f(x)=\frac{1}{2}\|A x-b\|_{2}^{2} . $$ Specialized linear algebra algorithms can solve this problem efficiently; however, we can also explore ...
Revolucion for Monica's user avatar
1 vote
1 answer
74 views

The preliminary of the Bandit Gradient Algorithm

In the papers introducing The Bandit Gradient Algorithm as Stochastic Gradient Ascent, the following relationship: is always considered as a preliminary and lacks proof for it. Does anyone know how ...
WilliamW's user avatar
1 vote
0 answers
81 views

RMSProp Momentum and Decay

I'm making an application of MobileNetV2 and according to their article: We train our models using TensorFlow. We use the standard RMSPropOptimizer with both decay and momentum set to 0.9. We use ...
Ricardo's user avatar
  • 11
1 vote
0 answers
13 views

Reinforcement learning with 0 rewards and costs

Suppose we have a hallway environment, i.e, $N$ nodes from left to right, and we can either move left or right. Moving left at the leftmost node does nothing and reaching the right most node gives you ...
Just_A_Doubt's user avatar
1 vote
0 answers
23 views

Searching for the underyling affine transformation in a ridge function

Quoting from Wikipedia: A ridge function is any function $f:\mathbb{R}^d\rightarrow\mathbb{R}$ that can be written as the composition of a univariate function with an affine transformation, that is: $...
Jonathan Jeffrey's user avatar
0 votes
0 answers
203 views

Coordinate descent for Lasso, Question about algorithm

I'm not sure why the algorithm computes $c_k$ with $\sum_{j \neq k} w_j x_{i, j}$. Why does one need to ignore the $k^{th}$ feature here? I'm not sure how this is derived. Is this the result of taking ...
user134661's user avatar
1 vote
1 answer
212 views

How does Gradient Descent treat multiple features?

As far as I know, when you reach the step, in a gradient descent algorithm, to calculate step_size, you calculate ...
alexandrosangeli's user avatar
1 vote
1 answer
22 views

SGD statistical guarantee

I have a question regard online learning with SGD. Is there a way to give a statistical guarantee that the value obtained after $n$ samples deviates at most $\epsilon$ from the real value? Thank you ...
Mark Regev's user avatar
2 votes
0 answers
55 views

In a machine learning system, why use differentially private SGD if our input data is already perturbed by a DP mechanism?

I'm trying to implement my own version of a deep neural network with differential privacy to preserve the privacy of the parties involved in the training dataset. I'm using the method by Abadi et al. ...
Saam's user avatar
  • 121
1 vote
0 answers
45 views

Backprop formula question

I'm reading this chapter https://www.deeplearningbook.org/contents/mlp.html of the Deep Learning book, and on page 209, they have this equation (assume there is no regularizer and no bias parameter): $...
nlzxz's user avatar
  • 11
1 vote
1 answer
134 views

Is Newton's algorithm really this much better than conjugate gradient descent?

I have a function I'm minimizing. I'm using conjugate gradient descent and the Newton algorithm. I am experiencing that the Newton algorithm is absurdly faster. Like, it finishes it 5-6 iterations, ...
seeit's user avatar
  • 13
1 vote
2 answers
827 views

Line-search does not guarantee convergence so how to use it?

Line-search/backtracking in gradient descent methods essentially boils down to picking the current estimate $\theta_n$ (which depends on the stepsize $\gamma$ and the prior estimate $\theta_{n-1}$) by ...
FiexPriceo's user avatar
0 votes
1 answer
448 views

What is the difference between derivative free optimization and derivative optimization in terms of advantages/disadvantages?

I understand the basic operation of the algorithms however i'm unclear as to when to use one over the other and what advantages/disadvantages they offer over each other. Also as an aside, if anyone ...
DIB98's user avatar
  • 13
1 vote
0 answers
92 views

Solving analytic gradient of loss function for neural networks [closed]

Please note that I am talking in about theory rather than ''what someone would do in a real, practical situation''. Given a multi-layer Perceptron with at least 1 hidden layer, and sigmoid (or other ...
SheppLogan's user avatar
0 votes
1 answer
53 views

Is SGD used in machine learning libraries?

SGD (Stochastic Gradient Descent) is used in most libraries of different programming languages. Is it also used in machine learning libraries?
hina munir's user avatar
2 votes
0 answers
154 views

How is momentum an approximation of Hessian based optimization?

In the answer to "what is the Hessian" at this site: https://stackoverflow.com/questions/23297090/how-calculating-hessian-works-for-neural-network-learning the person answering the question ...
seeker_after_truth's user avatar
2 votes
0 answers
33 views

Inverse kinematics step

I am working on an implementation of inverse kinematics using the jacobian transpose method. The implementation seems to be working as it does find the "theta" vector, although sometimes it might take ...
munHunger's user avatar
  • 121
1 vote
0 answers
83 views

Why don't Artificial Neural Networks Commonly Diverge?

Introduction: I'm using divergence here as to mean that the gradient is getting further and further from zero in stochastic gradient descent. I've written my own feed-forward neural network and tried ...
PolymorphismPrince's user avatar
2 votes
1 answer
102 views

Is there a universal learning rate for NeuralNetworks?

I'm currently creating a NeuralNetwork with backpropagation/gradient descent. There is this hyperparameter introduced called "learning rate" (η). Which has to be chosen to guarantee not overshooting ...
LU15.W1R7H's user avatar
2 votes
0 answers
63 views

About gradient descent on non-convex functions

There is this "folklore" result that gradient descent on a non-convex function takes $O(\frac n {\epsilon^2})$ steps to get to a point whose gradient norm is below $\epsilon$ and with SGD this takes $...
gradstudent's user avatar
0 votes
0 answers
50 views

Gradient Descent in MLPs using Computational Graphs

I'm working through Deep Learning by Goodfellow et al. The textbook introduces backpropagation for MLPs in page 203 (http://www.deeplearningbook.org/contents/mlp.html). However, it does not expand ...
Black's user avatar
  • 101
7 votes
0 answers
208 views

Algorithms for curve construction

I am interested in algorithms that construct continuous curves between two points in such a way that minimizes an energy functional of the curve. What sort of algorithms are most used for such tasks? ...
user3658307's user avatar
0 votes
0 answers
82 views

Gradient Descent With Constraints

I'm playing around with some historical stock data and attempting to optimize a portfolio. I essentially have created a function that generates certain statistics about a portfolio (right now it's ...
turnt's user avatar
  • 101
1 vote
2 answers
2k views

What are the challenges of using gradient descent on hyper parameters λ and η to find out their optimum values?

A question from chapter 3 of Michael Nielsen's [Neural Networks and Deep Learning]: It's tempting to use gradient descent to try to learn good values for hyper-parameters such as the regularization ...
Kartik chincholikar's user avatar
4 votes
1 answer
3k views

How to show that cross entropy is minimized?

This Question is taken from the book Neural Networks and DeepLearning by Michael Nielsen The Question: In a single-neuron ,It is argued that the cross-entropy is small if σ(z)≈y for all training ...
Kartik chincholikar's user avatar
2 votes
1 answer
2k views

Why do we use the log in gradient-based reinforcement algorithms?

I've been reading some papers on reinforcement learning. $$\Delta w=\frac{\partial ln\ p_w}{\partial w}r$$ I often see expressions, similar to the above one, where the weights (denoted by $w$) are ...
user avatar
2 votes
1 answer
847 views

Calculating gradient in a neural net using batches

I am a CS student learning about neural nets. Currently I am confused about how to train a neural net in batches. If I calculate error in a batch, I will get a vector of errors e.g. real1 - predicted1,...
swedishfished's user avatar
4 votes
1 answer
581 views

MDS minimization with gradient descent

I have the following multiple dimensional scaling (MDS) minimization problem in vectors $v_1, v_2, \dots, v_n \in \mathbb R^2$ $$\min_{v_1, v_2, \dots, v_n} \sum_{i,j} \left( \|v_i - v_j\| - d_{i,j} \...
CodeKingPlusPlus's user avatar
0 votes
0 answers
407 views

What is role of parameter learning rate, lr, and momentum constant, mc in Neural Networks?

can anyone describes the more simplified mathematical formulation of learning rate, lr, and momentum constant, mc in Neural Networks while training the data?
Case Msee's user avatar
  • 101
2 votes
0 answers
444 views

Lazy Stochastic Gradient Descent: Multiplicative vs Additive

I am reading Bob Carpenter's note at http://lingpipe.files.wordpress.com/2008/04/lazysgdregression.pdf and William Cohen's note at http://www.cs.cmu.edu/~wcohen/10-605/notes/sgd-notes.pdf. They ...
user59369's user avatar
3 votes
1 answer
3k views

Is it possible to solve the Mountain Car reinforcement learning task with linear Q-Learning using the state as direct input?

I'm trying to solve the Mountain Car task on OpenAI Gym (reach the top in 110 steps or less, having a maximum of 200 steps per episode) using linear Q-learning (the algorithm in figure 11.16, except ...
rcpinto's user avatar
  • 471
3 votes
1 answer
611 views

Why updating only a part of all neural network weights does not work?

I am having a problem with my program of deep neural network using Theano. In my deep neural network, I have several layers of neural network to predict an output given a certain input. Because of an ...
The Lazy Log's user avatar
4 votes
1 answer
2k views

Why update weights and biases after training a Neural Network on whole set of training samples

I am reading the book Neural Networks and Deep Learning by Micheal Nielsen. In the second chapter of his book, he describes the following algorithm for updating weights and biases for a neural ...
user5139637's user avatar
2 votes
0 answers
156 views

Computing $\mathrm{tr}(X^{-1}Y)$ efficiently

I know that one can compute the expression $X^{-1}\mathbf{v}$ quickly with conjugate gradient method. Is there a similar approach for computing $\mathrm{tr}(X^{-1}Y)$? Similarly interesting to me are ...
R S's user avatar
  • 129
0 votes
0 answers
1k views

Classifying responses into yes/no

So my problem is as follows: I get responses (such as "yeah whatever", "yes do it", "no don't do it", "nah", "yeah do it" etc.) and I need to classify them into either "yes" or "no" i.e. a binary ...
Mathguy's user avatar
  • 411
2 votes
1 answer
60 views

Determining if my artificial neural network needs additional layers

I have implemented a neural network for load forecasting in Microsoft Excel. My structure is very simplistic and involves only 1 hidden layer and 4 neurons. (See picture) I trained my network with a ...
stjenkins's user avatar
3 votes
0 answers
203 views

Speed up minimizing quadratic function by FFT

I'm trying to understand the following excerpt from a paper: Subproblem 1: computing $S$. The $S$ estimation subproblem corresponds to minimizing $$ \sum_{p}(S_p - I_p)^2 + \beta((\partial_xS_p -...
Yu Dai's user avatar
  • 131
3 votes
2 answers
4k views

Gradient descent overshoot - why does it diverge?

I'm thinking about gradient descent, but I don't get it. I understand that it can overshoot the minimum when the learning rate is too large. But I can't understand why it would diverge. Let's say we ...
user avatar
2 votes
2 answers
8k views

How to optimize a function by maximizing 1 variable and minimizing another?

Problem I want to implement an optimization algorithm for my file transfer program. The program buffers data in a local file before uploading to central server periodically and it compresses the ...
rajeshnair's user avatar
3 votes
1 answer
109 views

Mathematical optimization with thresholded optimization function

Gradient descent can be used to minimize an objective function $\Phi:\mathbb{R}^d \to \mathbb{R}$, if we know how to evaluate $\Phi$ on any input of our choice. However, my situation is a little ...
D.W.'s user avatar
  • 156k