does a non-terminated string count as a token in c?

Question

so, I am preparing for an exam which includes lexical analysis from compiler design. I was wondering what is the number of tokens in the following code-

int main() {
  /* comment
  printf("Hello */ There ");*/
  return 0;
}

so, I am thinking upto first "*/" it will be a multiline comment, so, after "there" a string will start without terminating. will the last string is counted as a token?

Answer 1

No, in C unterminated strings are not tokens. The C language definition precisely describes what a token is; among other things, they include (complete) string literals and a fallback category of "single non-whitespace characters" which are not otherwise matched by the lexical grammar. This latter category does not, however, include " and ' characters; if these characters are not matched as part of a longer token, the result is specifically flagged as undefined behaviour.

So your text is invalid precisely because it cannot be divided into tokens. A conforming implementation must respond by providing at least one diagnostic message.

All of the above is specific to the C language standard. It is not a "principle of computation" and should not be applied to any other programming language.

---

For additional precision, the C standard defines two categories of tokens. Initial program text analysis (in phases 1 and 2) splits the text into preprocessing tokens and whitespace (whitespace includes comments, which are not tokens). The translation process then passes the stream of preprocessing tokens through phases 3 to 6, commonly known as "the preprocessor" although it is an integral part of program translation.

Preprocessing leaves most tokens intact but there are features which allow adjacent tokens to be fused or converted into a string literal token. Also, macros may ignore their arguments causing those tokens to vanish. It is not possible to split a token into multiple tokens.

In phase 7, the preprocessing tokens which survive preprocessing must be converted to tokens. Although this is described as a conversion, no textual modification is made; what is converted is the category of the token. Not every character sequence which qualified as a preprocessing token can be treated as a token; phase 7 conversion of such a token will fail and a diagnostic message will be produced.

So it can make sense to talk about an "illegal token" (@, for example) which is still a token. But unterminated string and character literals do not all into that category. They really are not tokens at all.

See §5.1.1.2 of the C standard for a precise description of the translation phases. Tokens are defined in §6.4, which includes the prohibition on unmatched " and ' in paragraph 3:

A token is the minimal lexical element of the language in translation phases 7 and 8. The categories of tokens are: keywords, identifiers, constants, string literals, and punctuators. A preprocessing token is the minimal lexical element of the language in translation phases 3 through 6. The categories of preprocessing tokens are: header names, identifiers, preprocessing numbers, character constants, string literals, punctuators, and single non-white-space characters that do not lexically match the other preprocessing token categories. If a ' or a " character matches the last category, the behavior is undefined.