The set of unranked $Σ$-trees, denoted by $T$, is the smallest set of strings over $Σ$ and the parenthesis symbols ‘)’ and ‘(’ such that for each $a∈Σ$ and $w∈T^∗$, $a(w)$ is in $T$.
What we have here is an inductive definition. The base case is implicit because $\epsilon \in T^*$ even if $T = \emptyset$; making is explicit, the definition is (reading $a$ as free variable):
$\qquad\begin{align*}
&\ \phantom{\implies}\ a() \in T \\
w \in T^+ &\implies a(w) \in T
\end{align*}$
The term "smallest set" is established, but somewhat silly, since all these sets are infinite; we mean the minimal set. Formally speaking, it's the smallest fixed point of this inductive definition; see here and here for some more on that.
As for you question, imagine this definition unfolding in an infinite process:
- $a \in T \implies a(a) \in T$.
- $a(a) \in T \implies a(a(a)) \in T$.
- ... and so on...
All you can ever do is wrap things in parentheses; thus you can never creates mismatching pairs like you propose.
A formal proof would be an induction along the inductive definition, of course!
- Base case: all alphabet symbols have matching parentheses.
- Inductive step: taking elements of $T$ with matching parentheses, those that are "one step larger" have matching parentheses as well.