The tag has no usage guidance.

45 questions
Filter by
Sorted by
Tagged with
1 vote
135 views

### What does RSGD stand for?

I'm reading a paper that involves an algorithm for RSGD. It's clearly a form of stochastic gradient descent, but I can't find what the R stands for. The authors provide their own implementation of it, ...
1 vote
31 views

### Understanding gradient flow of a linearized wide neural network

I've been trying to fully understand the paper "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent" (available here), but I'm stuck on the linearization part, ...
66 views

### Create a simple Neural Network of n layers in python from scratch with numpy to solve XOR example problem using Batch Gradient Descent

I'm a young programmer that was interested by machine learning. I watched videos and read articles about the theory behind simple neural networks. However, I can't manage to set it up correctly. I've ...
55 views

### How to calculate the upper bound of the gredient of a multi layer ReLu neural network

Layers: We shall denote in the following the layer number by the upper script $\ell$. We have $\ell=0$ for the input layer, $\ell=1$ for the ﬁrst hidden layer, and $\ell=L$ for the output layer. The ...
21 views

### Convergence rate of quasi-newton method for non-convex objective function

Consider a real-valued $L$-smooth and non-convex objective function $f: \mathbb{R}^n \mapsto \mathbb{R}$. There exists a bound on number of iterations in order to find a (local) minima using ordinary ...
1 vote
105 views

### Why when a function is quadratic, the approximation by Newton's method is exact, and the algorithm converges to the global minimum in a single step?

Suppose we want to find the value of $x$ that minimizes $$f(x)=\frac{1}{2}\|A x-b\|_{2}^{2} .$$ Specialized linear algebra algorithms can solve this problem efficiently; however, we can also explore ...
1 vote
74 views

### The preliminary of the Bandit Gradient Algorithm

In the papers introducing The Bandit Gradient Algorithm as Stochastic Gradient Ascent, the following relationship: is always considered as a preliminary and lacks proof for it. Does anyone know how ...
1 vote
81 views

### RMSProp Momentum and Decay

I'm making an application of MobileNetV2 and according to their article: We train our models using TensorFlow. We use the standard RMSPropOptimizer with both decay and momentum set to 0.9. We use ...
1 vote
13 views

### Reinforcement learning with 0 rewards and costs

Suppose we have a hallway environment, i.e, $N$ nodes from left to right, and we can either move left or right. Moving left at the leftmost node does nothing and reaching the right most node gives you ...
1 vote
23 views

1 vote
134 views

### Is Newton's algorithm really this much better than conjugate gradient descent?

I have a function I'm minimizing. I'm using conjugate gradient descent and the Newton algorithm. I am experiencing that the Newton algorithm is absurdly faster. Like, it finishes it 5-6 iterations, ...
1 vote
827 views

### Line-search does not guarantee convergence so how to use it?

Line-search/backtracking in gradient descent methods essentially boils down to picking the current estimate $\theta_n$ (which depends on the stepsize $\gamma$ and the prior estimate $\theta_{n-1}$) by ...
448 views

### What is the difference between derivative free optimization and derivative optimization in terms of advantages/disadvantages?

I understand the basic operation of the algorithms however i'm unclear as to when to use one over the other and what advantages/disadvantages they offer over each other. Also as an aside, if anyone ...
1 vote
92 views

### Solving analytic gradient of loss function for neural networks [closed]

Please note that I am talking in about theory rather than ''what someone would do in a real, practical situation''. Given a multi-layer Perceptron with at least 1 hidden layer, and sigmoid (or other ...
53 views

### Is SGD used in machine learning libraries?

SGD (Stochastic Gradient Descent) is used in most libraries of different programming languages. Is it also used in machine learning libraries?
154 views

### How is momentum an approximation of Hessian based optimization?

In the answer to "what is the Hessian" at this site: https://stackoverflow.com/questions/23297090/how-calculating-hessian-works-for-neural-network-learning the person answering the question ...
33 views

### Inverse kinematics step

I am working on an implementation of inverse kinematics using the jacobian transpose method. The implementation seems to be working as it does find the "theta" vector, although sometimes it might take ...
1 vote
83 views

### Why don't Artificial Neural Networks Commonly Diverge?

Introduction: I'm using divergence here as to mean that the gradient is getting further and further from zero in stochastic gradient descent. I've written my own feed-forward neural network and tried ...