2
$\begingroup$

I am reading Programming Language Pragmatics by Michael Scott. He says that on a first pass, a compiler will break a program into a series of tokens. He says that it will check for malformed tokens, like 123abc or $@foo (in C).

What is a malformed token? A variable that does not meet the rules of variable-naming? An operator that does not exist (ex. "<-")?

Is this analogous to a misspelled word?

|cite|improve this question
$\endgroup$
  • $\begingroup$ That is any token that is rejected by a certain (languge-specific) finite state automaton. $\endgroup$ – MCH Aug 21 '12 at 17:29
3
$\begingroup$

The basic syntactic units in any textual programming language are tokens, which are, if you wish, the words in the programming language. The compiler parses these tokens to build sentences and so forth.

A malformed token is a string of characters that is not a valid word for the programming language.

It is not a variable that doesn't match the naming rules – not all tokens are variables. It is not an operator that is unknown – sometimes operators can be defined by the programmer. It is one of the smallest pieces of syntax that will never correspond to something sensible.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Aren't the basic syntactic units called lexemes and tokens are classes of lexemes? $\endgroup$ – saadtaame Aug 21 '12 at 20:47
  • $\begingroup$ Indeed, you are correct. $\endgroup$ – Dave Clarke Aug 21 '12 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.