Say i have code snippet like -->

m -= n;

is minus and assignment considered as a single token or they will be considered as different token?

So the total token count will be 4 or 5?

  • 5
    $\begingroup$ That depends entirely on how you implement your compiler and for what language. $\endgroup$
    – orlp
    Jul 12 '19 at 11:22
  • $\begingroup$ for c programming language $\endgroup$
    – user107500
    Jul 12 '19 at 11:24

In the C language you cant insert space between '-' and '=', so you are better to implement '-=' as the single lexeme.


This has mostly to do with programming language design, not compiler design.

As an example, the C language defines that “-=“ is a valid token, and it also says that always the longest possible sequence of characters is used, so in the C language we gave a “-=“ token and not a “-“ token followed by a “=“ token.

The compiler is free to do what it likes but must of course translate the language correctly, so 99.99999999% chances that it creates a single token.

There’s an interesting situation in C++ where << can be used for nested templates. Not sure how the tokeniser would handle it, but the compiler must follow the language rules.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy