Share Your Experience: Take the 2024 Developer Survey

The tag has no usage guidance.

46 questions
Filter by
Sorted by
Tagged with
306 views

### Find minimum of a function only knowing the ordering of a set of input points

Suppose I have a function $f: \mathbb{R}^n\rightarrow\mathbb{R}$. All I know about the function is, I have a set of pairs of vectors ($\vec{v}_a$, $\vec{v}_b$) for which I know which one is greater (i....
1 vote
137 views

### What does RSGD stand for?

I'm reading a paper that involves an algorithm for RSGD. It's clearly a form of stochastic gradient descent, but I can't find what the R stands for. The authors provide their own implementation of it, ...
• 125
1 vote
33 views

### Understanding gradient flow of a linearized wide neural network

I've been trying to fully understand the paper "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent" (available here), but I'm stuck on the linearization part, ...
93 views

### Create a simple Neural Network of n layers in python from scratch with numpy to solve XOR example problem using Batch Gradient Descent

I'm a young programmer that was interested by machine learning. I watched videos and read articles about the theory behind simple neural networks. However, I can't manage to set it up correctly. I've ...
63 views

### How to calculate the upper bound of the gredient of a multi layer ReLu neural network

Layers: We shall denote in the following the layer number by the upper script $\ell$. We have $\ell=0$ for the input layer, $\ell=1$ for the ﬁrst hidden layer, and $\ell=L$ for the output layer. The ...
27 views

### Convergence rate of quasi-newton method for non-convex objective function

Consider a real-valued $L$-smooth and non-convex objective function $f: \mathbb{R}^n \mapsto \mathbb{R}$. There exists a bound on number of iterations in order to find a (local) minima using ordinary ...
1 vote
139 views

### Why when a function is quadratic, the approximation by Newton's method is exact, and the algorithm converges to the global minimum in a single step?

Suppose we want to find the value of $x$ that minimizes $$f(x)=\frac{1}{2}\|A x-b\|_{2}^{2} .$$ Specialized linear algebra algorithms can solve this problem efficiently; however, we can also explore ...
1 vote
79 views

### The preliminary of the Bandit Gradient Algorithm

In the papers introducing The Bandit Gradient Algorithm as Stochastic Gradient Ascent, the following relationship: is always considered as a preliminary and lacks proof for it. Does anyone know how ...
• 51
1 vote
95 views

### RMSProp Momentum and Decay

I'm making an application of MobileNetV2 and according to their article: We train our models using TensorFlow. We use the standard RMSPropOptimizer with both decay and momentum set to 0.9. We use ...
• 11
1 vote
14 views

### Reinforcement learning with 0 rewards and costs

Suppose we have a hallway environment, i.e, $N$ nodes from left to right, and we can either move left or right. Moving left at the leftmost node does nothing and reaching the right most node gives you ...
1 vote
23 views

• 11
1 vote
195 views

### Is Newton's algorithm really this much better than conjugate gradient descent?

I have a function I'm minimizing. I'm using conjugate gradient descent and the Newton algorithm. I am experiencing that the Newton algorithm is absurdly faster. Like, it finishes it 5-6 iterations, ...
• 13
1 vote
992 views

### Line-search does not guarantee convergence so how to use it?

Line-search/backtracking in gradient descent methods essentially boils down to picking the current estimate $\theta_n$ (which depends on the stepsize $\gamma$ and the prior estimate $\theta_{n-1}$) by ...
508 views

### What is the difference between derivative free optimization and derivative optimization in terms of advantages/disadvantages?

I understand the basic operation of the algorithms however i'm unclear as to when to use one over the other and what advantages/disadvantages they offer over each other. Also as an aside, if anyone ...
• 13
1 vote
96 views

### Solving analytic gradient of loss function for neural networks [closed]

Please note that I am talking in about theory rather than ''what someone would do in a real, practical situation''. Given a multi-layer Perceptron with at least 1 hidden layer, and sigmoid (or other ...
• 133
54 views

### Is SGD used in machine learning libraries?

SGD (Stochastic Gradient Descent) is used in most libraries of different programming languages. Is it also used in machine learning libraries?
• 171
163 views

### How is momentum an approximation of Hessian based optimization?

In the answer to "what is the Hessian" at this site: https://stackoverflow.com/questions/23297090/how-calculating-hessian-works-for-neural-network-learning the person answering the question ...
36 views

### Inverse kinematics step

I am working on an implementation of inverse kinematics using the jacobian transpose method. The implementation seems to be working as it does find the "theta" vector, although sometimes it might take ...
• 121
1 vote
85 views

### Why don't Artificial Neural Networks Commonly Diverge?

Introduction: I'm using divergence here as to mean that the gradient is getting further and further from zero in stochastic gradient descent. I've written my own feed-forward neural network and tried ...