Lexical Analyzer mostly deletes comments and white-spaces. One example where I think Lexical Analyzer might not be discarding white-spaces is in Python language, as indentation has a important role in python. But I can't think of practical example where comments are/should be kept by Lexical Analyzer? Is there any such example you can think of?

  • 1
    $\begingroup$ "Lexical Analyzer mostly deletes comments and white-spaces." -- just to be safe: the core task of the lexer is something very different. Discarding unnecessary symbols is a side-effect. $\endgroup$ – Raphael Sep 25 '16 at 11:57
  • $\begingroup$ @Raphael - Can you please explain a bit. $\endgroup$ – Mayank Jain Sep 25 '16 at 13:54
  • $\begingroup$ What exactly is not clear? $\endgroup$ – Raphael Sep 25 '16 at 14:43
  • $\begingroup$ The lexical analyzer parses the program into lexical elements, such as number literal, identifier, the reserved word if, and so on. In some cases, it also stores auxiliary data, for example the value a number literal or the name of the identifier. When doing so, the lexical analyzer ignores comments and white-spaces (in most languages). $\endgroup$ – Yuval Filmus Sep 25 '16 at 17:03

White-space and comments can contain valuable information.

For white-space, you can look at layout sensitive languages, like Python or Haskell. I'm not sure how their lexers work, but off the top of my head I would store an indentation level, not the actual white-space.

For comments, it can be useful to store these if you want to generate documentation from Javadoc-style comments. For Javadoc, the parameter types do not have to be specified in the docblock (unlike PHPDoc, for instance). This means that a documentation generator would have to look at both the comments and the method header (everything before {). For the latter the lexer can be used, for the former it cannot (presuming the lexer removes comments).

I'm note sure how Javadoc-based documentation generators work, so perhaps the above is inaccurate, but the general idea is correct. I'm working on the Cloogle project, a Clean function search engine similar to Haskell's Hoogle. We hook into the Clean compiler to parse function types from all standard libraries. It would be nice if we could show comments around the definition as well, but the lexer removes this information. So, if we want to display comments, we have to adapt the compiler's frontend, or write a simplistic program that finds comments around some function, without lexing everything. If the lexer and parser would store comments, we would have a much more elegant solution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.