Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [lexical-analysis]

The tag has no usage guidance.

2
votes
1answer
49 views

Lexicographically smallest down-right path in matrix

Here is the problem which I thought was simple dynamic programming, which is however not the case. Given an $N \times M$ matrix of numbers from 1 to $NM$ (each number occurs only once), find a path ...
2
votes
1answer
19 views

Why are regular expression used in lexical analysis instead of grammars?

Even though grammars can do what regular expressions do, why are regular expression employed instead of grammars in lexical analysis?
0
votes
1answer
23 views

Token Scanner for programming Language(Lexical Analysis)

This DFA is a token scanner for a programming language.I would like to add keywords of the programming language(if,else,end ... etc) in the DFA so the lexical analyzer can recognize them. The ...
1
vote
1answer
86 views

How to implement a maximal munch lexical analyzer by simulating NFA or running DFA?

I'm planning to implement a lexical analyzer by either simulating NFA or running DFA using the input text. The trouble is, the input may arrive in small chunks and the memory may not be enough to hold ...
0
votes
1answer
121 views

Counting tokens in compilers, lexical analyser

Let's start with the question. Say I have a C language statements follows it 458cat 2.01 = 96.87abc a.2 ; -my question is how many tokens are there in the above statements. Secondly does white ...
0
votes
0answers
69 views

What token does a “peek” operation refer to in Lexical analysis?

Given a grammar for a space delimited list of words: S -> word { space word } word -> [a-zA-Z]+ space -> [ \t]+ And given the input "Hello World", ...
1
vote
1answer
415 views

Lexical analysis and reserved words

My book says lexical analysis detects reserved words. How can lexical analysis detect whether we are using reserved words in incorrect places ? For eg :- while(a>0) and int while = 5. How can ...
0
votes
1answer
41 views

Do we need regular expression first or finite state automata in lexical anlysing? [closed]

I'm a bit confused about the concept of finite state automata (FSA) and regular expression (RE) in lexical analysis. I have reading some books about compiler construction. At the part of tokenization, ...
0
votes
2answers
2k views

How does lexical analyzer remove white spaces from source file?

In compiler at phase lexical analyzer, how does lexical analyze strip out white spaces from source file?.
2
votes
3answers
456 views

Thompson's construction, transforming a regular expression into an equivalent NFA

There is this case of concatenation that says (from wikipedia): The initial state of N(s) is the initial state of the whole NFA. The final state of N(s) becomes the initial state of N(t). The final ...
3
votes
1answer
487 views

Why using finite automata in implementing lexical analyzers

I studied the subject of building a lexical analyzer using finite automata, through the classical way: Regular Expression -> NDA -> DFA. And I found it elegant, and a more solid approach for building ...
0
votes
1answer
226 views

How do hand-written lexers work?

Im studying a book called "Engineering a compiler" by Linda Torczon and Margaret Cooper. In the second chapter which discusses lexical analysis they mention three aproaches to writing a lexical ...
3
votes
1answer
1k views

White-space and comment by lexical analyzer

Lexical Analyzer mostly deletes comments and white-spaces. One example where I think Lexical Analyzer might not be discarding white-spaces is in Python language, as indentation has a important role in ...
3
votes
1answer
419 views

Why do we not use CFGs to describe the structure of lexical tokens?

This was an exam question for my course and I am struggling to actually answer it in a way that is not fluff. Here is my current answer: CFGs describe how non-terminal symbols are converted into ...
4
votes
2answers
436 views

What is the point of delimiters and whitespace handling

I see that language specifies reserved words, delimiters and whitespaces in the lexer section. Are delimiters just like reserved identifiers in the punctuation space or they have additioinal function ...
1
vote
1answer
189 views

What is token-type in Lexical analysis?

I'm currently studying compiler construction book "Compilers Principles, Techniques, and Tools (2nd Edition)" , in page unit 3.1 , page 113 , example 3.2 . i cannot understand what is this kind of ...
0
votes
1answer
178 views

Why the given lexical specification will not process the following strings?

I'm taking a Compiler MOOC on my own time. The class is self paced. An answer was given to a question but I can't understand the answer. In fact I'm not even sure how to interpret the question. Here ...
2
votes
1answer
62 views

Prefix computation used for lexical analysis?

I'm preparing a presentation on prefix computation (aka scan, the generalization of prefix summation to any associative operator) for a class I'm taking on parallel algorithms. Several lists of ...
0
votes
3answers
2k views

What data is stored in the symbol table for a number token?

I'm reading the Dragon Book. The following is from the start of Section 3.1.3. When more than one lexeme can match a pattern, the lexical analyzer must provide the subsequent compiler phases ...
1
vote
2answers
98 views

Trying to understand a token file for lexical analysis

I am reading this article about compiler. I am facing some problem in understanding the content of the token file. Specifically, what is the meaning of the following lines: ...
6
votes
4answers
680 views

Are there real lexers that use NFAs directly instead of first transforming them to DFAs?

I am taking the Coursera class on compilers and in the lesson about lexers it is hinted that there is a time-space tradeoff between using non-deterministic finite automaton (NFA) and deterministic ...