I'm currently studying compiler construction book "Compilers Principles, Techniques, and Tools (2nd Edition)" , in page unit 3.1 , page 113 , example 3.2 .

i cannot understand what is this kind of method ?

E = m * 2;

conversion to token types(this method) below

<id , E> <op=> <id , m> <op*> <number , 2> <op;>

it is token-type pairs ? or is there any technical term associated ?

  • $\begingroup$ When you say "this kind of method", what kind of method are you talking about? I'm not sure what you're referring to. Can you edit the question to make the question self-contained? I don't have the book and can't look at that page of the book, so I can't tell what you are asking about. $\endgroup$
    – D.W.
    Sep 29, 2015 at 23:06

1 Answer 1


I don't know what is the proper name for what you're seeing, but it is the standard way to lexically analyze text. You divide it into tokens of specific types. For the sake of context-free parsing (the next step in the parsing chain), you only need the type of each lexeme; but further steps down the road will need to know the semantic content (sometimes called annotation) of each lexeme. For example, the context-free parser doesn't care which number $2$ is; it only needs to know that it's a number. Further down the road, it will suffice to know that this number is an integer. Even further, you would need to know that this integer is specifically $2$.

Some lexemes are atomic, for example the various operators (though in principle we could group, say, + and - together). We need to separate them because syntactically = is different from +, and + is different from * (due to operator precedence). These don't need any annotation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.