The problem is to find the minimum of the cost defined by,
$$\sum_{k=0}^3 x_k^2 + u_k^2$$ with the initial condition $x_0=5$ and relation $0\le x_{k+1}=x_k+u_k\le5$.
$$ x_0=5\ \stackrel{u_0=-3}{\xrightarrow{\hspace{2em}}}\ x_1=2\ \stackrel{u_1=-1}{\xrightarrow{\hspace{2em}}}\ x_2=1\ \stackrel{u_2=-1}{\xrightarrow{\hspace{2em}}}\ x_3=0\ \stackrel{u_3=0}{\xrightarrow{\hspace{2em}}} $$
The cost of the state transition above is
$$(5^2+(-3)^2) + (2^2+(-1)^2)+ (1^2+(-1)^2) + (0^2+0^2)=5^2+16=41$$
Here is the simple arithmetic that shows 41 is the minimum cost. Note that a square number is always non-negative.
- If $x_1=0,1,4,5$, then $u_0=-5, -4, -1, 0$. $$\ \sum_{k=0}^3 x_k^2 + u_k^2\ge x_0^2+(u_0^2+x_1^2)\ge 5^2 + 17$$
- If $x_1=3, x_1=2$, then $u_0=-2, -3$. $$\ \sum_{k=0}^3 x_k^2 + u_k^2= 5^2 + 4 + 9 + (u_1^2+x_2^2 + u_2^2 + x_3^2+u_3^2)\\=5^2 + 13 + (u_1^2+x_2^2 + u_2^2 + x_3^2)\ge 5^2 + 13 + 3=5^2+16,$$ where the last inequality holds as among the numbers $u_1, x_2, u_2$, either all of them are $1$ or $-1$, or one of them is bigger than $1$ or smaller than $-1$, i.e., a number whose square is no less than $2^2=4$. $\checkmark$
Tracing the proof above, we can see that the optimal control sequence is $u_0^*=-3$, $u_1^*=-1$, $u_2^*=-1$, $u_3^*=0$.