# White-space and comment by lexical analyzer

Lexical Analyzer mostly deletes comments and white-spaces. One example where I think Lexical Analyzer might not be discarding white-spaces is in Python language, as indentation has a important role in python. But I can't think of practical example where comments are/should be kept by Lexical Analyzer? Is there any such example you can think of?

• "Lexical Analyzer mostly deletes comments and white-spaces." -- just to be safe: the core task of the lexer is something very different. Discarding unnecessary symbols is a side-effect. – Raphael Sep 25 '16 at 11:57
• @Raphael - Can you please explain a bit. – Mayank Jain Sep 25 '16 at 13:54
• What exactly is not clear? – Raphael Sep 25 '16 at 14:43
• The lexical analyzer parses the program into lexical elements, such as number literal, identifier, the reserved word if, and so on. In some cases, it also stores auxiliary data, for example the value a number literal or the name of the identifier. When doing so, the lexical analyzer ignores comments and white-spaces (in most languages). – Yuval Filmus Sep 25 '16 at 17:03

For comments, it can be useful to store these if you want to generate documentation from Javadoc-style comments. For Javadoc, the parameter types do not have to be specified in the docblock (unlike PHPDoc, for instance). This means that a documentation generator would have to look at both the comments and the method header (everything before {). For the latter the lexer can be used, for the former it cannot (presuming the lexer removes comments).