0
$\begingroup$

Say i have code snippet like -->

m -= n;

is minus and assignment considered as a single token or they will be considered as different token?

So the total token count will be 4 or 5?

$\endgroup$
2
  • 5
    $\begingroup$ That depends entirely on how you implement your compiler and for what language. $\endgroup$
    – orlp
    Jul 12 '19 at 11:22
  • $\begingroup$ for c programming language $\endgroup$
    – user107500
    Jul 12 '19 at 11:24
1
$\begingroup$

In the C language you cant insert space between '-' and '=', so you are better to implement '-=' as the single lexeme.

$\endgroup$
0
$\begingroup$

This has mostly to do with programming language design, not compiler design.

As an example, the C language defines that “-=“ is a valid token, and it also says that always the longest possible sequence of characters is used, so in the C language we gave a “-=“ token and not a “-“ token followed by a “=“ token.

The compiler is free to do what it likes but must of course translate the language correctly, so 99.99999999% chances that it creates a single token.

There’s an interesting situation in C++ where << can be used for nested templates. Not sure how the tokeniser would handle it, but the compiler must follow the language rules.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy