Your instructor is right, and this seems like an odd choice.
$O(\infty)$ arises usually as a way to say "this algorithm may never terminate, it's awful, let's never use it."
Big-O and Running Time
First, what about big-O complexity of your problem? Well, problems don't have big-O complexity, algorithms do. So if there is no algorithm to solve your problem, then we can't talk about how many steps it takes an algorithm to solve your problem.
Complexity Classes, P, NP, etc.
But, what about complexity classes, NP-hard, etc?
The main classes are restricted to decidable problems. For example, $NP$ is the class of problems solveable by a Turing Machine running in non-deterministic polynomial time. But this is a subset of the class of solveable problems. Similar statements can be made about $P$, $EXP$, $PSPACE$, etc.
What about "hardness?" Can an undecidable problem be $NP$-hard? Yes, but not in an interesting way.
Informally, to show that a problem is $NP$-hard, we assume that we have an algorithm solving it, and we show that we can then solve another $NP$-hard problem in polynomial time with a polynomial number of calls to our algorithm.
But in your case, to do so is to assume that there is an algorithm solving an unsolveable problem. This means we've assumed a contradiction, so we can derive a polynomial time algorithm for some NP-hard problem, since we can derive literally anything from the contradiction. We've assumed False, and this provides us with no useful information.
The idea of standard complexity isn't even well-defined for an undecidable problem.
You problem will be in basically no complexity classes, but it will be Hard for almost every complexity class.