In compiler at phase lexical analyzer, how does lexical analyze strip out white spaces from source file?.
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ It just ignores it. What kind of answer are you looking for? $\endgroup$– David RicherbyCommented Oct 10, 2017 at 8:28
-
$\begingroup$ It's a finite automaton. Go figure. $\endgroup$– RaphaelCommented Oct 10, 2017 at 11:10
-
2$\begingroup$ This question seems to have an answer in any resource on lexers. What research have you done? What did you not understand? $\endgroup$– RaphaelCommented Oct 10, 2017 at 11:10
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
16
A brief answer is "by ignoring them". A compiler may scan the entire code more than once, but of course at least once in which case we call it a single-pass compiler. The task of lexical analyzer (or sometimes called simply scanner) is to generate tokens. This is done simply by scanning the entire code (in linear manner by loading it for example into an array) from the beginning to the end symbol-by-symbol and grouping them into tokens. Thus, while checking each symbol (of the source code) you just skip the white-space symbols. Similarly you skip comments in the source code.
As to my personal experience, I had to implement a single-pass compiler for a made-up simple language which allowed both white space symbols (TAB, NEW-LINE, SPACE) and comments. I loaded the entire source code into a linear array and as my lexical analyzer went over the code character by character it simply skipped all white space symbols and comments by moving forward the pointer pointing to the current character.
But depending on the implementation you could first pass over the source code just to remove all white spaces, and then on the second pass generate tokens.
Caution: "ignoring whitespace" shouldn't be interpreted as "remove all white space symbols from the source and then further process the source code". In some cases depending on the language construct white spaces may indicate when a token ends for example in the following piece of code
int foo(int x)
{
return x+2;
}
The tokens TYPE (int) and the IDENTIFIER (foo) are separated by a single SPACE. But the IDENTIFIER (foo) and immediate LEFTPAR are not separated by any white space symbol.
Also (on Derek Elkins' comment) real compilers usually use whitespace for producing clear error messages, for example for keeping track of line numbers.
answered Oct 10, 2017 at 6:43
-
$\begingroup$ Nitpick: "ignoring whitespace" could be read as if the lexer would yield the same result on the input string if all whitespace was removed. That's typically not the case: whitespace is used to detect were tokens end. "Ignore" here means that whitespace is not translated into tokens by itself. $\endgroup$– RaphaelCommented Oct 10, 2017 at 11:12
-
$\begingroup$ @Raphael we do not always use white-spaces to detect end of tokens and in general white spaces do not indicate where token ends. That's not the case. Consider the following piece of code
a=123;. We have the following tokens: IDENTIFIER, EQUAL, INT, SEMICOLON. The first three tokens are not separated by any whitespace symbol. Though there are cases when two tokens are separated by white spaces, for examplevoid int func(int x).... $\endgroup$ Commented Oct 10, 2017 at 11:46 -
1$\begingroup$ Yes. The general rule is that the set of token classes must form a prefix-free set, roughly speaking. (Precedence rules, reserved keywords, etc muddle the situation.) Whitespace (as something that is not part of any token) is a useful option to effect this, and weeaken the rule somewhat. In fact, I can't think of many languages that make do completely without whitespace. $\endgroup$– RaphaelCommented Oct 10, 2017 at 13:10
-
1$\begingroup$ Raphael: I don't know how to interpret "roughly speaking" in that comment, but to a first approximation there is no requirement at all that token classes be prefix-free, and very few languages (if any) manage that. Practically all token sets will include tokens which are suffixes of tokens in other sets:
==vs.=;forestvs.for. Numbers are practically the only exception. The normal rule is "maximal munch", which explicitly allows sets without the prefix property. But even that rule is occasionally circumvented. (3..4is a range in some languages, for example. Or<::in C++.) $\endgroup$– riciCommented Oct 10, 2017 at 15:08 -
1$\begingroup$ @Zephyr I guess this is the same as what operation in $2*3+1$ we should perform first: multiplication or or addition. We usually first do multiplication then addition. In other words, we resolve this disambiguity by introducing operator precedence. So the answer to your question depends on a language design and its implementation. In C/C++ I guess it would be first
a++and then addition as you suggest. Just try and see how it is evaluated in C/C++. $\endgroup$ Commented Oct 16, 2017 at 21:08