This problem can be tidied up using the floor function, where $\lfloor x\rfloor$ denotes the smallest integer greater than or equal to $x$. For example, we have $\lfloor 3.14\rfloor=3$ and $\lfloor 6\rfloor=6$. In particular, if $n$ and $m$ are positive integers, then $\lfloor n/m\rfloor$ is the quotient you obtain when you divide $n$ by $m$ (i.e., throw away the remainder).
Then, for integers $q>0$ you have a family of languages
$$
L_q=\{a^nb^m\mid \lfloor n/m\rfloor=q\}
$$
We'll generalize your proof slightly to show that $L_q$ is not regular for any integer $q>0$. For a fixed $q$ assume that $L_q$ was regular. Then the Pumping Lemma implies that there is an integer $p$ such that any string $w\in L_q$ of length greater than or equal to $p$ can be written as $w=xyz$ with
- $xy^iz\in L_q$ for any $i\ge 0$
- $|y|>0$
- $|xy|\le p$
Choose the string $a^{pq}b^p\in L_q$ and write $a^{pq}b^p=xyz$ as above. Then, as you noted, we may say $xy=a^k$ without loss of generality (the case where $xy=b^k$ will be handled in the same way as below and the case where $xy=a^ib^j$ will produce an immediate contradiction for $xy^2z$, since it will be the wrong form to be in $L_q$).
Now we know that $y=a^j$ for some $0<j\le p$. The first inequality comes from condition (2) of the PL and the second comes from condition (3). As you noted, we'll then have
$$
xyz=(a^r)(a^j)(a^{pq-r-j}b^p)
$$
and so
$$
xz=(a^r)(a^{pq-r-j}b^p)=a^{pq-j}b^p
$$
Now we'll show that $xz\notin L_q$, contradicting condition (1) of the PL.
If, to the contrary, $xz\in L_q$, we'd have to have
$$
\left\lfloor\frac{pq-j}{p}\right\rfloor=q
$$
but
$$
\left\lfloor\frac{pq-j}{p}\right\rfloor=\left\lfloor q-\frac{j}{p}\right\rfloor
$$
But now we came to the heart of your question: since $0<j\le p$ by (2) and (3) we'll have $q-1\le q-(j/p) < q$, so
$$
\left\lfloor\frac{pq-j}{p}\right\rfloor=q-1
$$
and so $xz\notin L_q$, giving us the contradiction we needed, completing the proof that $L_q$ is not regular for any $q>0$.