What you see there is a common regexp syntax.
[ \t\n] matches any character that is a space, a tab or a new line.
[a-zA-Z_][a-zA-Z0-9_]* matches any string of length at least one consisting of roman letters, digits and underscore that does not start with a digit.
[0-9]+\.[0-9]* matches any number with at least one digit before the (mandatory) period.
[0-9]+ matches any number with at least one digit.
SAVE_TOKEN; return X likely tells the compiler generator that when the preceeding expression matches, create a token of type
X and assign it the matching string.
The input string is processed from left to right and whenever one of the expressions matches the current buffer, the directive associated with the expression is executed and the buffer emptied.
Note that there is no
SAVE_TOKEN directive for the whitespace rule, so all whitespaces are removed. All other matches generate appropriate tokens that can later be used in a (context-free) grammar to parse the program.
Note that one (common, but arbitrary) convention is that rules take precedence from top to bottom. This does not take effect here as the expressions are mutually exclusive.