2
$\begingroup$

In a fully connected Neural Network, each perceptron has it's bias term $b$ which is learnt. Often (example, in Linear/ Logistic Regression), when we don't want to treat this bias term explicitly, we add an extra "constant" feature $x_0 = 1$ to each training example. Can we show whether this will/ will not work in a Neural Network? i.e. we add an $x_0 = 0$ term to each training example but don't have a bias term in the perceptrons.
My idea is that such a Neural Network should be representationally weaker than the standard one as the term $x_0 = 1$ added will "dissolve" after the first layer. I tried constructing simple examples like (I used activation $g(x) = 1$ if $x \geq 0$ else $0$, at the hidden layer and output layer outputs, for a classification task)

  1. $1$ hidden layer with $1$ perceptron and $1$ perceptron in the output layer. $x \in \Re^2$ (before adding $x_0 = 1$)
  2. $1$ hidden layer with $2$ perceptrons and $1$ perceptron in the output layer. $x \in \Re^2$ (before adding $x_0 = 1$)

In both cases, the set of Decision Boundaries drawn by both types of NN's were same.

I used this activation for simplicity, you're free to use any. Can someone help me with a simple enough example to show that the "proposed" Neural Network will be weaker than the one where each perceptron has it's bias term. Thanks!

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.