In NLP, does the lexer have to tag the tokens before the parser?

I.e. does the lexer have to classify the tokens to morphological categories before the parser?

I'm thinking yes, but is this also the only way to do the parsing?

  • 1
    $\begingroup$ NLP is a huge area, with lots of similar-but-not-the-same techniques. Which one are you targeting, specifically? $\endgroup$ – vonbrand Dec 26 '15 at 23:04
  • $\begingroup$ @vonbrand I'm just looking for yes/no answers. I don't know of the techniques, but I'm trying to construct a very rudimentary NLP "pipeline". To me it seems that I would have to tag the tokens before the parser, but I'm interested in knowing, whether there are alternative techniques. $\endgroup$ – mavavilj Dec 26 '15 at 23:11
  • $\begingroup$ No blanket yes/no answers possible when there are dozens, if not hundereds, of different approaches. $\endgroup$ – vonbrand Dec 30 '15 at 12:16

Morphological tags can help the parser. On the other hand, the complete sentence structure, maybe even the paragraph context may help to finally disambiguate possible tags for a token. So there is no yes/no answer. Except, maybe, that tagging is, afaik, usually not attributed as work of the lexer, but rather a module of its own.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.