Say i have code snippet like -->
m -= n;
is minus and assignment considered as a single token or they will be considered as different token?
So the total token count will be 4 or 5?
2 Answers
$\begingroup$
$\endgroup$
1
In the C language you cant insert space between '-' and '=', so you are better to implement '-=' as the single lexeme.
-
$\begingroup$ This is asinine. C standard is only an authoritative source of truth for an implementation of C. $\endgroup$– Ivan CCommented Jun 25, 2022 at 10:54
$\begingroup$
$\endgroup$
This has mostly to do with programming language design, not compiler design.
As an example, the C language defines that “-=“ is a valid token, and it also says that always the longest possible sequence of characters is used, so in the C language we gave a “-=“ token and not a “-“ token followed by a “=“ token.
The compiler is free to do what it likes but must of course translate the language correctly, so 99.99999999% chances that it creates a single token.
There’s an interesting situation in C++ where << can be used for nested templates. Not sure how the tokeniser would handle it, but the compiler must follow the language rules.
