New answers tagged


Several angles here. The distinction between "token" and "syntax" is somewhat arbitrary. Both can (and have been) described by CFGs. One reason for the split is that we read e.g. a C program in terms of tokens (variable names, operators, reserved words, comments), not in term of characters. So it makes sense for the compiler writer to do what comes ...


Regular expressions are recognized by finite state machines, while context free grammar recognized by push-down automata. Push-down automata is not efficient as FSM in terms of time, hence it's all about efficiency.


Scanner (tokenizer) will be enough. Most programming languages do not allow comment nesting, so there is no need to use recursive parsers to strip all comments.

Top 50 recent answers are included