so, I am preparing for an exam which includes lexical analysis from compiler design. I was wondering what is the number of tokens in the following code-

int main() {
  /* comment
  printf("Hello */ There ");*/
  return 0;

so, I am thinking upto first "*/" it will be a multiline comment, so, after "there" a string will start without terminating. will the last string is counted as a token?


No, in C unterminated strings are not tokens. The C language definition precisely describes what a token is; among other things, they include (complete) string literals and a fallback category of "single non-whitespace characters" which are not otherwise matched by the lexical grammar. This latter category does not, however, include " and ' characters; if these characters are not matched as part of a longer token, the result is specifically flagged as undefined behaviour.

So your text is invalid precisely because it cannot be divided into tokens. A conforming implementation must respond by providing at least one diagnostic message.

All of the above is specific to the C language standard. It is not a "principle of computation" and should not be applied to any other programming language.

For additional precision, the C standard defines two categories of tokens. Initial program text analysis (in phases 1 and 2) splits the text into preprocessing tokens and whitespace (whitespace includes comments, which are not tokens). The translation process then passes the stream of preprocessing tokens through phases 3 to 6, commonly known as "the preprocessor" although it is an integral part of program translation.

Preprocessing leaves most tokens intact but there are features which allow adjacent tokens to be fused or converted into a string literal token. Also, macros may ignore their arguments causing those tokens to vanish. It is not possible to split a token into multiple tokens.

In phase 7, the preprocessing tokens which survive preprocessing must be converted to tokens. Although this is described as a conversion, no textual modification is made; what is converted is the category of the token. Not every character sequence which qualified as a preprocessing token can be treated as a token; phase 7 conversion of such a token will fail and a diagnostic message will be produced.

So it can make sense to talk about an "illegal token" (@, for example) which is still a token. But unterminated string and character literals do not all into that category. They really are not tokens at all.

See § of the C standard for a precise description of the translation phases. Tokens are defined in §6.4, which includes the prohibition on unmatched " and ' in paragraph 3:

A token is the minimal lexical element of the language in translation phases 7 and 8. The categories of tokens are: keywords, identifiers, constants, string literals, and punctuators. A preprocessing token is the minimal lexical element of the language in translation phases 3 through 6. The categories of preprocessing tokens are: header names, identifiers, preprocessing numbers, character constants, string literals, punctuators, and single non-white-space characters that do not lexically match the other preprocessing token categories. If a ' or a " character matches the last category, the behavior is undefined.

  • $\begingroup$ so, in other words, any code which has a "Compilation error" can not be tokenized? $\endgroup$ – john chakder Jan 2 '20 at 16:37
  • 2
    $\begingroup$ @john: not what I said. There are lots of error conditions which are detected after tokenising. $\endgroup$ – rici Jan 2 '20 at 16:39
  • 1
    $\begingroup$ The compilation process consists of scanning (tokenizing), parsing, context-sensitive analysis, IR code conversion, and instruction generation. Therefore, there could be many sources to a "compilation error". Unterminated string literal could count towards an error while forming tokens and the compiler may choose to report it and stop or it may make a complete pass through the code and also report other errors. For example, you may also see ‘There’ undeclared as another error that would typically be reported much later after the scanning stage. $\endgroup$ – ss09 Jan 2 '20 at 18:11
  • $\begingroup$ @ss09: that particular sequence doesn't take into account languages like C (and many others) whose definitions include macro expansion. In particular, in C a macro argument may be a token which cannot be converted in phase 7; that's completely acceptable if the macro does something which avoids the need to convert the token (such as turn it into a character string or just not use it). But there is no way to treat a single " as a token, because of the explicit prohibition I mention in the answer. So the original question is actually relevant to correct parsing of a C program text. $\endgroup$ – rici Jan 2 '20 at 20:51
  • $\begingroup$ @rici I overlooked this important fact. Thank you for pointing it out and also for your detailed answer. $\endgroup$ – ss09 Jan 3 '20 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.