I was reading the book "Computer systems: A programmer's perspective" and saw that the operator, "~", is a NOT operator. When I use it to NOT positive integers, it returns the negative of the original number plus one. Why does that happen?
That's because computers use $2$'s complement representation of negative numbers. $5$ in binary is $101$. Assume an $8$-bit architecture. Then $5$ would be represented as $00000101$. Hence ~$5$ will flip all the bits which gives $11111010$ which is the $1$'s complement of $5$. Adding $1$ to it gives you the $2$'s complement representation which the computer interprets as $-5$. Hence ~$5=11111010$ is $-5-1=-6$.