# Why separate lexing and parsing?

It's possible to parse a document using a single pass from a state machine. What is the benefit of having two passes, ie. having a lexer to convert text to tokens, and having a parser to test production rules on those tokens? Why not have a single pass that applies production rules directly to the text?

• – User Feb 28 '15 at 9:22
• This has already been discussed on CS,stackexchange, with many very technical comments in an answer to Expressive power of lexer + parser. But there may be room there for further answers. – babou Feb 28 '15 at 12:52
• I wonder if pipeline-style parallelism (albeit highly imbalanced stages) might be a side advantage. Both instruction and data cache behavior might also be interesting. How much (if at all) such would reduce compile time would depend on the specific hardware. – Paul A. Clayton Feb 28 '15 at 13:41
• One fairly obvious (at least to me) reason is that you can then use the scanner tool separately. In practice, I frequently use flex to scan input, but seldom need the full power of yacc. – jamesqf Feb 28 '15 at 19:01

• Having one pass instead of two passes involves closure properties. If you consider that lexers are transducers belonging to a formal family $T$, that can be combined with parsers belonging to a formal family $P$, you must wonder whether composing them stays in the same family $P$, or requires a more complex formal family $P_T$, thus requiring different algorithmics, and possibly more difficult tuning of syntax. Parsing technology often relies on specialized families (such as LR or LL variants) that may not have the right closure properties. – babou Feb 28 '15 at 12:45