6
$\begingroup$

Can someone please point me towards a rigorous derivation of the energy function of a discrete Hopfield network. What I want, is the derivation must start out with the structure of the network and prove that the critical points of the dynamical system can be obtained by minimization of some function (possibly by constructing the Lyapunov function of the dynamical system). I cannot seem to find such a rigorous proof. Thankyou!

$\endgroup$
6
$\begingroup$

I believe what you want to show is that the energy function is monotonically decreasing from time $t$ to time $t+1$ given the state update rules. Since there is only a finite number of states, this means the state must converge to a equilibrium under the given dynamics.

To this end let $\mathbf{s}(t) \in \{0,1\}^n$ be the state of the network at time $t$. We update the network state by picking a unit $s_i(t)$ and updating the state according to $$ s_i(t+1) = \left\{\begin{array}{r l} 1:& \sum_{j} w_{ij} s_j(t) > 0\\ 0:& \text{otherwise}\end{array}\right. $$ and define the energy function as $$ E(\mathbf{s}(t)) = -\frac{1}{2}\sum_i\sum_jw_{ij}s_i(t)s_j(t), $$ where $w_{ij}$ is the weight between unit $i$ and $j$, $w_{ij} = w_{ji}$ and $w_{ii} = 0$ for al $i$.

WLOG assume we will update the $i^{th}$ unit. First, because of the restrictions on the weights, we may rewrite $E(\mathbf{s}(t))$ as the difference of 2 terms $$ E(\mathbf{s}(t)) = -\frac{1}{2}\sum_{j\neq i}\sum_{k\neq i}w_{jk}s_j(t)s_k(t) - s_i(t)\sum_jw_{ij}s_j. $$ Let $c_t = -\frac{1}{2}\sum_{j\neq i}\sum_{k\neq i}w_{jk}s_j(t)s_k(t)$ be the part of the energy function that does not depend on $s_i(t)$. Now if $s_i(t+1) = s_i(t)$ then clearly we have $E(\mathbf{s}(t+1)) = E(\mathbf{s}(t))$, so we only need to consider the 2 other cases.

Case 1: If $s_i(t+1) = 1$ and $s_i(t) = 0$ then it must be that $\sum_{j} w_{ij} s_j(t) = \Delta_t > 0$. So $$ E(\mathbf{s}(t+1)) = c_{t+1} - \Delta_t = c_t - \Delta_t < c_t = E(\mathbf{s}(t)). $$ Case 2: If $s_i(t+1) = 0$ and $s_i(t) = 1$ then it must be that $\sum_{j} w_{ij} s_j(t) = \Delta_t \le 0$. So $$ E(\mathbf{s}(t+1)) = c_{t+1} = c_t \le c_t - \Delta_t = E(\mathbf{s}(t)). $$ Having considered each case we can conclude that $E(\mathbf{s}(t+1)) \le E(\mathbf{s}(t))$ and is therefore monotonically decreasing.

$\endgroup$
  • $\begingroup$ Thanks for the answer. I will examine the proof. My initial attempt was to derive the energy function for the discrete net from the continuous case by considering the additive model of a neuron. But I couldnt figure out how to discretize the eqns and impose some condition on the energy function to convert it into a discrete one. In that process, when the eqn is discretized, the form of the energy function doesnt remain the same and my attempts failed. Any help in this direction will be appreciated! $\endgroup$ – swarnim_narayan Jul 9 '13 at 17:40
  • $\begingroup$ This proof is very understandably written! +1 It is intended for the asynchronous update rule, I assume, because it assumes $\forall j \neq i . s_j (t) = s_j(t+1)$. How does this proof generalize to a synchronous update model where every node is updated simultaneously? $\endgroup$ – lucidbrot Jan 25 '18 at 7:43
0
$\begingroup$

One way to go about it is to consider the energy function for a (fully visible) Boltzmann machine which approaches a hop field net as T->0. You can derive the sigmoidal activation function from the energy via the boltzmann distribution and Bayes rule P(output=1| net input) for each unit.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.