1
$\begingroup$

I am building an analyzer that takes in text and numbers the lines; taking in

hello how are you
i'm fine thanks

as input and using the code:

%{
    int i=0;
%}
%option noyywrap
%%
.* {i++;printf("[%d] %s \n",i,yytext);} 
%%
void main(){
    yylex();
}

this yields:

[1] hello how are you 

[2] i'm fine thanks 

there's an empty line there, why? using this code fixes the issues:

%{
    int i=0;
%}
%option noyywrap
%%
[ \t]*\n
.* {i++;printf("[%d] %s \n",i,yytext);} 
%%
void main(){
    yylex();
}

this yields:

[1] hello how are you 
[2] i'm fine thanks 

I am confused about why there is an empty line using the first code, and why adding

[ \t]*\n

fixes the problem.

$\endgroup$

1 Answer 1

2
$\begingroup$

There are two things you need to know about Lex (and hence Flex, which follows the same semantics):

  1. It buffers a line of input at a time.
  2. The dot . symbol includes the new line/carriage return at the end of a line.

The first point is the important one. It means that Lex tokens cannot span a line break. If you need to handle multi-line tokens (e.g. a programming language with multi-line strings or multi-line comments), this is typically handled with start states.

Your input file (assuming Unix line endings; if you're using DOS line endings, mentally replace \n with \r\n), is essentially this string:

"hello how are you\ni'm fine thanks\n"

When Lex reads, this, it starts by buffering the first line:

"hello how are you\n"

The whole line matches the regular expression .*, and in particular, the \n character matches the . symbol.

So when you print the token text:

printf("[%d] %s \n",i,yytext);

what you are actually printing is:

"[1] hello how are you\n \n"

That is why you see a blank line in the output. Adding the other rule gives Lex a higher-priority token to consume the line ending.

(Note: Examples follow, but I have not actually tested any of these!)

This would also have worked:

[^\r\n]*  {i++;printf("[%d] %s \n",i,yytext);}
[\r\n]    ;

Here, the first line tells Lex that you want to match everything except line endings, and the second line gives Lex a rule to consume the line ending.

So would using the lookahead operator /:

.*/[\r\n]*  {i++;printf("[%d] %s \n",i,yytext);}
[\r\n]      ;

If you really want the whole line, not including the line ending, it's also good practice to include explicit start-line and end-line operators:

^[^\r\n]*$  {i++;printf("[%d] %s \n",i,yytext);}
[\r\n]      ;

(By the way, I put an explicit semicolon on rules for which I want Lex to do nothing. That's just my style, but I think it's a good habit to be in if you do a lot of Lex.)

Lex buffers input lines because it was designed in the 1970s when the memory and address space of computers was much smaller than it is today. It was also designed for early Unix, where everything is a small tool which takes text as input and output, to be connected together with pipes.

Modern thinking among compiler writers is that it makes more sense just to memory-map the whole input file and let the operating system do the buffering for you. This way, you don't have to worry about how line buffering interacts with your lexical analyser.

$\endgroup$
2
  • $\begingroup$ wouldn't just using [^\r\n]* {i++;printf("[%d] %s \n",i,yytext);} be enough? using it alone without [\r\n] produces the same empty line. the code: .*/[\r\n]* {i++;printf("[%d] %s \n",i,yytext);} [\r\n] ; produces : warning, dangerous trailing context and the code: ^[^\r\n]*$ {i++;printf("[%d] %s \n",i,yytext);} [\r\n] ; produces: [1] hello how are you i'm fine thanks //missing number here// $\endgroup$
    – AlexG
    Commented Feb 21, 2023 at 12:47
  • $\begingroup$ Probably. It's been a while since I used Lex in anger and, as noted, this code is untested. $\endgroup$
    – Pseudonym
    Commented Feb 21, 2023 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.