Questions tagged [gradient-descent]

The tag has no usage guidance.

16 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7
votes
0answers
178 views

Algorithms for curve construction

I am interested in algorithms that construct continuous curves between two points in such a way that minimizes an energy functional of the curve. What sort of algorithms are most used for such tasks? ...
3
votes
0answers
112 views

Speed up minimizing quadratic function by FFT

I'm trying to understand the following excerpt from a paper: Subproblem 1: computing $S$. The $S$ estimation subproblem corresponds to minimizing $$ \sum_{p}(S_p - I_p)^2 + \beta((\partial_xS_p -...
2
votes
0answers
20 views

Inverse kinematics step

I am working on an implementation of inverse kinematics using the jacobian transpose method. The implementation seems to be working as it does find the "theta" vector, although sometimes it might take ...
2
votes
0answers
45 views

About gradient descent on non-convex functions

There is this "folklore" result that gradient descent on a non-convex function takes $O(\frac n {\epsilon^2})$ steps to get to a point whose gradient norm is below $\epsilon$ and with SGD this takes $...
2
votes
0answers
313 views

Lazy Stochastic Gradient Descent: Multiplicative vs Additive

I am reading Bob Carpenter's note at http://lingpipe.files.wordpress.com/2008/04/lazysgdregression.pdf and William Cohen's note at http://www.cs.cmu.edu/~wcohen/10-605/notes/sgd-notes.pdf. They ...
2
votes
0answers
148 views

Computing $\mathrm{tr}(X^{-1}Y)$ efficiently

I know that one can compute the expression $X^{-1}\mathbf{v}$ quickly with conjugate gradient method. Is there a similar approach for computing $\mathrm{tr}(X^{-1}Y)$? Similarly interesting to me are ...
1
vote
0answers
22 views

How is momentum an approximation of Hessian based optimization?

In the answer to "what is the Hessian" at this site: https://stackoverflow.com/questions/23297090/how-calculating-hessian-works-for-neural-network-learning the person answering the question ...
1
vote
0answers
61 views

Why don't Artificial Neural Networks Commonly Diverge?

Introduction: I'm using divergence here as to mean that the gradient is getting further and further from zero in stochastic gradient descent. I've written my own feed-forward neural network and tried ...
1
vote
1answer
1k views

Gradient descent overshoot - why does it diverge?

I'm thinking about gradient descent, but I don't get it. I understand that it can overshoot the minimum when the learning rate is too large. But I can't understand why it would diverge. Let's say we ...
0
votes
0answers
13 views

What is the difference between derivative free optimization and derivative optimization in terms of advantages/disadvantages?

I understand the basic operation of the algorithms however i'm unclear as to when to use one over the other and what advantages/disadvantages they offer over each other. Also as an aside, if anyone ...
0
votes
0answers
8 views

Clarification on descending direction in optimization of function

Could someone clarify for me why given $f:\mathbb{R}^n \rightarrow\mathbb{R}$ to optimize an iterative function according to : $p^k=-M\nabla f(x^k)$ for $p^k$ to be descending direction the matrix M ...
0
votes
0answers
32 views

Gradient Descent in MLPs using Computational Graphs

I'm working through Deep Learning by Goodfellow et al. The textbook introduces backpropagation for MLPs in page 203 (http://www.deeplearningbook.org/contents/mlp.html). However, it does not expand ...
0
votes
0answers
59 views

Gradient Descent With Constraints

I'm playing around with some historical stock data and attempting to optimize a portfolio. I essentially have created a function that generates certain statistics about a portfolio (right now it's ...
0
votes
0answers
359 views

What is role of parameter learning rate, lr, and momentum constant, mc in Neural Networks?

can anyone describes the more simplified mathematical formulation of learning rate, lr, and momentum constant, mc in Neural Networks while training the data?
0
votes
0answers
854 views

Classifying responses into yes/no

So my problem is as follows: I get responses (such as "yeah whatever", "yes do it", "no don't do it", "nah", "yeah do it" etc.) and I need to classify them into either "yes" or "no" i.e. a binary ...
-1
votes
0answers
8 views

regression DNN: gradient checking doesn't match with backpropagation derivatives

I think the problem lies within a bad implementation of back propagation, it's the only way i could explain this (gradient checking that doesn't match backprop derivatives), but i'm not able to find ...