Let's start with the question. Say I have a C language statements follows
it 458cat 2.01 = 96.87abc a.2 ;
-my question is how many tokens are there in the above statements. Secondly does white space like tabs, newline, makes a token or not?
- If you are interested in my solution then proceed on your own risk. Firstly I think in lexical analyser there is no such thing as white space so after removal of that the above statement would look catastrophic so, I have not taking that approach.
- Now we see 'it' as an identifier similarly, '458' as integer, 'cat' as identifier, (please not that I may have mistaken here which i don't know that is '458cat' may be invalid identifier so the lexical analyser may report an error, but I don't know will that happen or not. you may argue that what I have written is wrong because if it was true then the identifier like '678hello' will be a valid identifier but since its not then iam wrong. But take a different approach like if all white space between '468 abc' being vanished (this is what lexical analyser does first) then it would like '468abc' . Initially in case of 468 abc we were taking 468 as integer and abc as valid identifier but after white space removal things are changed so what I think is it will be reported in syntax analyser but I don't know that am I right), similarly, '2.01' as real integers, '=' as operator, '96.87' as real number, 'abc' as identifier, 'a ' as identifier, .2 as real number and ' ; ' as special symbol.
- I most certainly know that iam not right but if you could correct where I was wrong it will be helpful