If you do not flip the kernel, you simply obtain a different operation that is called cross correlation. When the filter is symmetric, like a Gaussian, or a Laplacian, convolution and correlation coincides. But when the filter is not symmetric, like a derivative, you get different results.
The reason why convolution is preferred over correlation is that it has nicer mathematical properties. In particular, convolution is associative, while correlation in general is not.
For a more technical explanation we need to go into the frequency domain.
The main theorem of convolutions states that the Fourier tansform of the convolution $f*g$ of two functions $f$ and $g$ is equal (up to a constant depending on the transformation) to the product of the Fourier transforms of the two functions. In symbols
$$ {\mathcal {F}}\{f*g\}=k\cdot {\mathcal {F}}\{f\}\cdot {\mathcal {F}}\{g\}$$
where $\mathcal {F}$ is the Fourier transform. In the case of correlation, you would get multiplication by the complex conjugate, that is less nice, and in particular not associative.
Another interesting property of convolution is that convolving a kernel with
a unit impulse (e.g. a matrix with a single 1 at its center and 0 otherwise),
you get the kernel itself as a result. Correlation would flip the kernel,
instead.