Possible mistake in a book regarding parsing and lexical analysis

Question

I was just reading a book (Algorithms and Theory of Computation Handbook, Volume 1) and I came across the following passage :

"From a practical point of view, for each grammar G = (Σ,V, S, P) representing some language, the following two problems are important:

1. Membership problem: Given a string over Σ, does it belong to L(G)?
2. Parsing problem: Given a string in L(G), how can it be derived from S? The importance of the membership problem is quite obvious—given an English sentence or computer program, we wish to know if it is grammatically correct or has the right format. Solving the membership problem for context-free grammars is an integral step in the lexical analysis of computer programs, namely the stage of decomposing each statement into tokens, prior to fully parsing the program. For this reason, the membership problem is also often referred to as lexical analysis (cf. [6])."

I, by no means, am an expert. However, it's obvious to me that the author is confusing lexical analysis and parsing (syntax analysis). Answering the membership problem is exactly why we build parsers for compilers. And, tokenizing a program has nothing to do whatsoever with the grammar of the programming language.

In the past I have been very wrong about things like this, so I wanted to hear your views on this.

Thanks!

Answer 1

That statement does seem like a mistake. Context-free grammars are more relevant to parsing, whereas lexing is mainly driven by regular expressions, as evidenced by parser generators such as yacc and lexer generators such as lex.

Perhaps it used to be a paragraph about both regular expressions and context free grammars (given that the chapter does talk about both) that got moved around and merged with other stuff and was missed during later editing.