In a fully connected Neural Network, each perceptron has it's bias term $b$ which is learnt. Often (example, in Linear/ Logistic Regression), when we don't want to treat this bias term explicitly, we add an extra "constant" feature $x_0 = 1$ to each training example. Can we show whether this will/ will not work in a Neural Network? i.e. we add an $x_0 = 0$ term to each training example but don't have a bias term in the perceptrons.
My idea is that such a Neural Network should be representationally weaker than the standard one as the term $x_0 = 1$ added will "dissolve" after the first layer. I tried constructing simple examples like (I used activation $g(x) = 1$ if $x \geq 0$ else $0$, at the hidden layer and output layer outputs, for a classification task)

  1. $1$ hidden layer with $1$ perceptron and $1$ perceptron in the output layer. $x \in \Re^2$ (before adding $x_0 = 1$)
  2. $1$ hidden layer with $2$ perceptrons and $1$ perceptron in the output layer. $x \in \Re^2$ (before adding $x_0 = 1$)

In both cases, the set of Decision Boundaries drawn by both types of NN's were same.

I used this activation for simplicity, you're free to use any. Can someone help me with a simple enough example to show that the "proposed" Neural Network will be weaker than the one where each perceptron has it's bias term. Thanks!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.