1
$\begingroup$

ReLU units are said to be necessary in CNNs to introduce non-linearity which convolution does not involve. This is needed, because many real-world forms of data are non-linear.

What does non-linearity refer to?

$\endgroup$
1
$\begingroup$

A neural network, in its simplest form, consists of layers of neurons.

The first layer of neurons are the input, and the last layer forms the output.

A neuron $x$ on layer $i+1$ is connected to several neurons $y_1,\ldots,y_d$ on layer $i$, and it computes the following function: $$ x = f(\alpha_1 y_1 + \cdots + \alpha_d y_d + \beta), $$ where $f$ is a fixed function (nowadays often ReLU), and $\alpha_1,\ldots,\alpha_d,\beta$ are parameters which are learned in the training period. (The output layer often uses a different $f$.)

If $f$ is the identity function $f(z) = z$, then each neuron in the output is just an affine function of the input. This means that no matter how many layers the neural network has, all it can do is compute linear functions.

In order to allow the network to compute more complicated functions, we introduce nonlinearity, in the form of a non-linear function $f$ (that is, a function which is not of the form $f(z) = az + b$), such as ReLU, which is $f(x) = \max(x,0)$. This makes the neural network more expressive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.