Here are several interpretations of your question:
$L_1$ consists of all descriptions of Turing machines which are deciders (always halt) and the language that they decide is in NP.
This is undecidable, by reduction from the halting problem. Given a Turing machine $T$, construct a new Turing machine $M$ which erased its input and transfers control to $T$. Then $\langle M \rangle \in L_1$ iff $T$ halts on the empty input.
Indeed, if $T$ halts then $L(M) = \Sigma^*$, while if $T$ doesn't halt, then $M$ is not a decider.
$L_2$ consists of all descriptions of Turing machines which are either not deciders, or the language that they decide is in NP.
This is undecidable, by reduction from the halting problem. Given a Turing machine $T$, construct a new Turing machine $M$ which simulates $T$ on the empty tape, and if $T$ halts, solves some NEXP-complete problem on the original input. Then $\langle M \rangle \in L_2$ iff $T$ doesn't halt on the empty input.
Indeed, if $T$ halts on the empty input then $M$ decides an NEXP-complete problem, which by the nondeterministic time hierarchy theorem doesn't belong to NP. In contrast, if $T$ doesn't halt on the empty input, then $M$ is not a decider.
$L_3$ is the promise problem in which the input is a description of a Turing machine which is promised to be a decider, and the goal is to determine whether the language it decides is in NP.
Alternatively, you can ask whether there exists any decidable language $L_4$ such that
- The description of any machine deciding a language in NP belongs to $L_4$.
- The description of any machine deciding a language not in NP doesn't belong to $L_4$.
This is undecidable, in the sense that there is no Turing machine that halts and answers YES on all YES instances, and halts and answers NO on all NO instances.
Given a Turing machine $T$, construct a new Turing machine $M$ with two inputs $n,x$. The Turing machine simulates $T$ for $n$ steps. If $T$ halts within these $n$ steps, then it solves some NEXP-complete problem on $x$. Otherwise, it simply returns YES. The new machine $M$ is a YES instance of $L_3$ iff $T$ doesn't halt on the empty input.
Indeed, if $T$ halts on the empty input then there exists an NEXP-complete language $L$ and an integer $n$ such that $(n,x) \in L(M)$ iff $x \in L$, and so $L(M) \notin \mathsf{NP}$. In contrast, if $T$ doesn't halt on the empty input then $L(M)$ consists of all pairs $(n,x)$, and so $L(M) \in \mathsf{NP}$.
IsInNP
? $\endgroup$ – Andrej Bauer Sep 3 '19 at 7:39