In NLP, does the lexer have to tag the tokens before the parser?
I.e. does the lexer have to classify the tokens to morphological categories before the parser?
I'm thinking yes, but is this also the only way to do the parsing?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ NLP is a huge area, with lots of similar-but-not-the-same techniques. Which one are you targeting, specifically? $\endgroup$– vonbrandDec 26, 2015 at 23:04
-
$\begingroup$ @vonbrand I'm just looking for yes/no answers. I don't know of the techniques, but I'm trying to construct a very rudimentary NLP "pipeline". To me it seems that I would have to tag the tokens before the parser, but I'm interested in knowing, whether there are alternative techniques. $\endgroup$– mavaviljDec 26, 2015 at 23:11
-
$\begingroup$ No blanket yes/no answers possible when there are dozens, if not hundereds, of different approaches. $\endgroup$– vonbrandDec 30, 2015 at 12:16
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Morphological tags can help the parser. On the other hand, the complete sentence structure, maybe even the paragraph context may help to finally disambiguate possible tags for a token. So there is no yes/no answer. Except, maybe, that tagging is, afaik, usually not attributed as work of the lexer, but rather a module of its own.
