If an optimization problem is mixed integer non-convex problem. The optimal solution by applying brute exhaustive search is infeasible to be applied in practice due to high complexity i.e. $O(N^K)$. Can we say that it's NP hard problem? If yes, can someone explain with more details.
No, you can't infer NP-hardness from such an algorithm. Indeed, why should the existence of a slow algorithm mean that there can't be a fast algorithm?
For example, consider the problem of sorting $n$ integers. Suppose the first algorithm I give you is a one that tries all $n!$ permutations. Can we now safely conclude that this problem is somehow hard? Of course not and we know that there are many much faster polynomial-time algorithms for the problem.