Let us prove a more general result:
For each $m \geq 2$ there is a language $L$ such that $L,L^2,\ldots,L^{m-1}$ are not regular but $L^m$ is regular.
The language we construct will be unary, that is, of the form $L = \{a^n : n \in S\}$, where $S \subseteq \mathbb{N}$ is
$$
S = \{ km : k \geq 0 \} \cup \{ m^k + 1 : k \geq 1 \}.
$$
Let $1 \leq p \leq m-1$, and consider $p \cdot S$, i.e., the set of all sums of $p$ elements from $S$. Since all elements in $S$ are equivalent to either $0$ or $1$ modulo $m$, if $a \in S$ is equivalent to $p$ modulo $m$ then it must have the form
$$a = m^{k_1} + 1 + \cdots + m^{k_p} + 1. $$
The base $m$ representation of $a$ thus has at most $p+1$ non-zero digits (it could have fewer if some of the $k_i$ are equal). It is easy to check that a vanishing fraction of numbers have this property.
Let $L_{m,p} = \{ a^{km+p} : k \geq 0\}$. It follows that $L' = L^p \cap L_{m,p}$ is an infinite language with vanishing density, and so it cannot be regular (since regular unary languages are eventually periodic, and so if they have vanishing density they must be finite). Since $L_{m,p}$ is regular, it follows that $L^p$ isn't regular.
In contrast, $m \cdot S$ is cofinite — it contains all integers which are at least $m^2-1$ — and so $L^m$ is regular.
The language constructed above has the "regularity profile" $\{k : k \geq m\}$. This is the set of powers $k \geq 1$ such that $L^k$ is regular ($L^0$ is always regular). This prompts the following question:
What regularity profiles are possible?
Any regularity profile is closed under addition, but are there any other constraints?