# Trying to understand a token file for lexical analysis

I am reading this article about compiler. I am facing some problem in understanding the content of the token file. Specifically, what is the meaning of the following lines:

    [ \t\n]                 ;
[a-zA-Z_][a-zA-Z0-9_]*  SAVE_TOKEN; return TIDENTIFIER;
[0-9]+\.[0-9]*          SAVE_TOKEN; return TDOUBLE;
[0-9]+                  SAVE_TOKEN; return TINTEGER;


What is * and + at the end of the expressions ?

Thanks in advance for explaining me.

• Have you heard of regular expressions? If not, you should read about them. – D.W. Aug 26 '13 at 5:15

* means that any number (including zero) of preceding expression can occur

+ means that at least one instance of preceding expression must occur

• Thanks for the info. The article says "The first token tells us to skip all whitespace.", where is such a token? – gpuguy Aug 25 '13 at 10:47
• this one [ \t\n] ; – Bartosz Przybylski Aug 25 '13 at 12:58

What you see there is a common regexp syntax.

• [ \t\n] matches any character that is a space, a tab or a new line.
• [a-zA-Z_][a-zA-Z0-9_]* matches any string of length at least one consisting of roman letters, digits and underscore that does not start with a digit.
• [0-9]+\.[0-9]* matches any number with at least one digit before the (mandatory) period.
• [0-9]+ matches any number with at least one digit.
• SAVE_TOKEN; return X likely tells the compiler generator that when the preceeding expression matches, create a token of type X and assign it the matching string.

The input string is processed from left to right and whenever one of the expressions matches the current buffer, the directive associated with the expression is executed and the buffer emptied. Note that there is no SAVE_TOKEN directive for the whitespace rule, so all whitespaces are removed. All other matches generate appropriate tokens that can later be used in a (context-free) grammar to parse the program.

Note that one (common, but arbitrary) convention is that rules take precedence from top to bottom. This does not take effect here as the expressions are mutually exclusive.