Let's start with the question. Say I have a C language statements follows

  1. it 458cat 2.01 = 96.87abc a.2 ;

    -my question is how many tokens are there in the above statements. Secondly does white space like tabs, newline, makes a token or not?

  2. If you are interested in my solution then proceed on your own risk. Firstly I think in lexical analyser there is no such thing as white space so after removal of that the above statement would look catastrophic so, I have not taking that approach.
  3. Now we see 'it' as an identifier similarly, '458' as integer, 'cat' as identifier, (please not that I may have mistaken here which i don't know that is '458cat' may be invalid identifier so the lexical analyser may report an error, but I don't know will that happen or not. you may argue that what I have written is wrong because if it was true then the identifier like '678hello' will be a valid identifier but since its not then iam wrong. But take a different approach like if all white space between '468 abc' being vanished (this is what lexical analyser does first) then it would like '468abc' . Initially in case of 468 abc we were taking 468 as integer and abc as valid identifier but after white space removal things are changed so what I think is it will be reported in syntax analyser but I don't know that am I right), similarly, '2.01' as real integers, '=' as operator, '96.87' as real number, 'abc' as identifier, 'a ' as identifier, .2 as real number and ' ; ' as special symbol.
  4. I most certainly know that iam not right but if you could correct where I was wrong it will be helpful
  • 1
    $\begingroup$ I'm voting to close this question as off-topic because it appears to be specific to the C programming language. It may be on-topic at Stack Overflow. $\endgroup$ – David Richerby Sep 10 '18 at 14:38
  • $\begingroup$ @david: there are many who would say it's off-topic in SO because it has nothing to do programming, being an implementation detail of a particular compiler. However, tokenisation is specified by the C standard and it is has an effect on writing preprocessor macros (in certain obscure constructs) so it would fit into the language-lawyer tag. I fear that type of discussion wouldnot be at a level consistent with OP's expectations. Anyway, it's been asked before and answered (by me, amongst others) so even on-topic it would be a dupe. $\endgroup$ – rici Sep 10 '18 at 15:39
  • 1
    $\begingroup$ In short, although it is evidently a homework problem in certain CS courses, it seems to fall into a blackhole in SE. It's a question; it has an objective answer; people studying CS want to know the answer, but... $\endgroup$ – rici Sep 10 '18 at 15:43

In C, 458cat is a single ppnumber token. It's not a valid number, so it will eventually produce an error message, but it is tokenised as a single token.

There's a longer explanation of this behaviour in this StackOverflow answer.

  • $\begingroup$ Well thanks for answering but my point is what I could change 458cat to 888889aprilfool $\endgroup$ – Noob Sep 10 '18 at 14:39
  • $\begingroup$ @low_burning: it would still be a single ppnumber. Maybe I don't understand what you are asking. $\endgroup$ – rici Sep 10 '18 at 15:13
  • $\begingroup$ Firstly thanks for support and secondly I have understood and thirdly rici if I knew that there is a question that exists already I would have never asker it. Fourthly you always get an answer but through stack exchange it's pretty quick and by self it takes times and many reasoned assertion to reach a verdict and that is why I ask here. But you keep doing what you are doing. I wished I could have an up vote for you $\endgroup$ – Noob Sep 10 '18 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.