Simple example of pseudo-random sequence that is not periodic:
concatenate together the binary representations of all positive integers, in order:
110111001011101111000...
(Prepend a "." and it's called the binary Champernowne constant.)
Of course this isn't very high quality as far as pseudo-random sequences go,
but it demonstrates that it's possible without using much memory.
In this example, the memory needed to store the state is theoretically unbounded;
however it grows very slowly
(compared to, say, the memory needed to compute digits of $\pi$ or of $\sqrt{2}$).
Likewise for the time needed to generate each bit.
The unbounded memory requirement isn't a problem for a turing machine,
and it's probably not a problem in practice, either, since the growth is so slow,
but it depends on what you intend to use this thing for.
For instance, if all you're doing is actually generating bits in sequence,
then you can easily generate arbitrarily many bits of this aperiodic sequence,
on a real computer, without ever running out of memory in the lifetime of the universe.
However for that use case aperiodicity has no advantage over simply
having a very large period such as $2^{128}$, which won't be reached in the lifetime
of the universe either. So, not much difference from the point of view of "normal" usage.
The differentiator, I suppose, is if you want to do things like derive other PRNGs
from your PRNG, say by taking a subsequence at regular intervals.
If you start with the aperiodic PRNG, you'll get an aperiodic result no matter what period
you sample at, whereas if you start with a PRNG of period $2^{128}$, you'll start running into trouble
if the sampling period is sufficiently large.