If you check the definitions, big-O means "bounded from above" (with a fudging constant $c$ multiplying it all, starting at a high enough $n$), you can disregard "constants multiplying" and "slower growing terms"; so you can say:
$\begin{equation*}
5 n^2 \log n - \dfrac{100 n^2}{\log_4 n^2} + 40
= O(n^2 \log n)
\end{equation*}$
(could also say $O(n^3)$, or $O(n^4 \log^{10} n)$ for that matter, bound from above).
For a lower bound, the same works fine:
$\begin{equation*}
5 n^2 \log n - \dfrac{100 n^2}{\log_4 n^2} + 40
= \Omega(n^2 \log n)
\end{equation*}$
For a lower bound, $\Omega(1)$ or $\Omega(n \log n)$ work fine too.
As you have the same upper/lower bound:
$\begin{equation*}
5 n^2 \log n - \dfrac{100 n^2}{\log_4 n^2} + 40
= \Theta(n^2 \log n)
\end{equation*}$
Finding values of $c$ and $N_0$ I leave to you. They can be different for $O$ and $\Omega$!