Binary cross Entropy derivative?

Question

I am just learning backpropagation algorithm for NN and currently I am stuck with the right derivative of Binary Cross Entropy as loss function.

Here it is:

def binary_crossentropy(y, y_out):
    return -1 * (y * np.log(y_out) + (1-y)*np.log(1-y_out))

def binary_crossentropy_dev(y, y_out):
    return binary_crossentropy(y, y_out) * (1 - binary_crossentropy(y, y_out))

def binary_crossentropy_dev2(y, y_out):
    return (y_out - y)/ (y_out * (1-y_out))

But when comparing both derivative above with real numbers, their results are differenct altough they should be the same. What am I doing wrong with the above equations? Thank you!

Answer 1

Here is the definition of cross-entropy for Bernoulli random variables $\operatorname{Ber}(p),\operatorname{Ber}(q)$, taken from Wikipedia: $$ H(p,q) = p \log \frac{1}{q} + (1-p) \log \frac{1}{1-q}. $$ This is exactly what your first function computes.

The partial derivative of this function with respect to $p$ is $$ \frac{\partial H(p,q)}{\partial p} = \log \frac{1}{q} - \log \frac{1}{1-q} = \log \frac{1-q}{q}. $$ The partial derivative of this function with respect to $q$ is $$ \frac{\partial H(p,q)}{\partial q} = -\frac{p}{q} + \frac{1-p}{1-q} = \frac{(1-p)q-p(1-q)}{q(1-q)} = \frac{q-p}{q(1-q)}. $$ This is exactly what your third function computes.

I'm not sure what your second function computes. Also, there is no reason to expect that the partial derivative with respect to one variable will be the same as the partial derivative with respect to another variable.