This Question is taken from the book Neural Networks and DeepLearning by Michael Nielsen

The Question: In a single-neuron ,It is argued that the cross-entropy is small if σ(z)≈y for all training inputs. The argument relied on y being equal to either 0 or 1. This is usually true in classification problems, but for other problems (e.g., regression problems) y can sometimes take values intermediate between 0 and 1. Show that the cross-entropy is still minimized when σ(z)=y for all training inputs. When this is the case the cross-entropy has the value:

C is the cost function

σ is the sigmoid function

y is the expected output of the network

I am not sure if I understand the question correctly.If σ(z)=y wouldn't the partial derivative of C with respect to the weights(or Bias) be equal to zero?This will imply that the weight is already at the correct value and the cost function cannot be reduced any further by changing the weight.

enter image description here

z is the weighted input to the neuron.(z=wx+b)

Where b is the bias and x is the activation of a neuron in the previous layer.(In this case it is the input to the network itself)

w is the weight value

  • 1
    $\begingroup$ Welcome to CS.SE! Can you define all notation? What is $\sigma$? (sigmoid activation function?) What are $y$ and $z$? What is $C$? If you are copying from an assignment, I encourage you to properly credit the source of the material. Where did the equation at the bottom of your question come from? $\endgroup$
    – D.W.
    Apr 19 '17 at 22:00

You are calculating the so-called binary cross-entropy. Let $f(\cdot)$ be a sigmoid function. The binary cross-entropy between $y$ and $f(t)$ is

$$ F(t,y) = H(y,f(t)) = -y\log f(t) - (1-y)\log(1-f(t)). $$

When you compute the partial derivative of $F(t,y)$ with respect to $t$, you get:

$$ \frac{\partial F(t,y)}{\partial t} = f(t) - y $$

You reach the minimum when $f(t)=y$. In case of multiple variables, let:

$$ F(\vec{t},\vec{y}) = \frac{1}{n}\sum_{i=1}^n F(t_i,y_i) $$


$$ \frac{\partial F(\vec{t},\vec{y})}{\partial t_i} = \frac{f(t_i) - y_i}{n} $$

Now, if $t_i=\sum_j w_{ij}x_j+b_i$ is the activation, with the chain rule:

$$ \frac{\partial F(\vec{t},\vec{y})}{\partial w_{ij}}=\sum_k\frac{\partial F(\vec{t},\vec{y})}{\partial t_k}\cdot\frac{\partial t_k}{\partial w_{ij}}=\frac{1}{n}\cdot x_j\cdot[f(t_i) - y_i]$$

  • 1
    $\begingroup$ Next time, please use LaTeX. $\endgroup$ Apr 20 '17 at 8:10
  • $\begingroup$ Great. Didn't know latex is supported here. Edited with a bit more latex for clarity. $\endgroup$
    – user172818
    Apr 20 '17 at 13:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.