In NLP, does the lexer have to tag the tokens before the parser?
I.e. does the lexer have to classify the tokens to morphological categories before the parser?
I'm thinking yes, but is this also the only way to do the parsing?
Computer Science Stack Exchange is a question and answer site for students, researchers and practitioners of computer science. It only takes a minute to sign up.Sign up to join this community
Morphological tags can help the parser. On the other hand, the complete sentence structure, maybe even the paragraph context may help to finally disambiguate possible tags for a token. So there is no yes/no answer. Except, maybe, that tagging is, afaik, usually not attributed as work of the lexer, but rather a module of its own.