0
$\begingroup$

I know a lot of computer languages and they all use tokens. E.g. in very early BASIC you could say LET answer = 42, which is composed of seven tokens, LET, answer, =, 42, and three space tokens.

It seems that every character must be part of some token, including comments, which can be considered free-form tokens after the initial identifier.

Are there any languages completely or partially without tokens? Is it even possible for such a language to exist? What would it look like?

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ What would it mean for a language to be without tokens? I can always interpret each individual character as a token of its own, right? $\endgroup$
    – D.W. ♦
    Commented Sep 20, 2017 at 1:02
  • $\begingroup$ @D.W. No, the token 42 can't be thought of as a 4 token and a 2 token. The two characters together make an atomic token, i.e. the meaning is different if they are split. $\endgroup$
    – CJ Dennis
    Commented Sep 20, 2017 at 1:10
  • 2
    $\begingroup$ I argue that this is a matter of interpretation; I can define my programming language to have any syntax and semantics I want. Can you be more specific about what it would mean for a language to be without tokens? Can you give us a definition of what the word token means for you? $\endgroup$
    – D.W. ♦
    Commented Sep 20, 2017 at 1:16
  • 1
    $\begingroup$ The token is an atomic value to pass some piece of information, even if you restrict tokens to chars, where e.g. the ASCII value is used - it has some value, otherwise there is no way to give input to the parser. $\endgroup$
    – Evil
    Commented Sep 20, 2017 at 1:18
  • 1
    $\begingroup$ Consider a number in E-notation, e.g. -6.02E-23. How many tokens does it have? $\endgroup$
    – svick
    Commented Sep 20, 2017 at 8:55

1 Answer 1

Reset to default
4
$\begingroup$

I agree with the commenters that your question is either ill-defined or nonsensical. It all hinges on what you mean by "tokens", and what it would mean to "not have tokens."

If you mean multi-character sequences that are "chunked" together into larger words, consider Brainfuck — or actually most esolangs, including my favorite, Befunge — where every "command" is a single character. However, on the other hand, many Brainfuck interpreters actually treat [-] as a single "chunk" meaning "set the current tape cell to zero", so in that sense maybe you'd call [-] a "token".

On the other hand, if you consider single characters to be "tokens", then maybe we need to go the other direction, and find programming languages in which everything kind of flows together without divisions. The first thing that pops to mind is Inform 7 (yes, that's Inform 7 source code!), where meaning is conveyed through English prose. Obviously Inform 7 has a parser that deals in words, but those words don't directly correspond to anything like LET in Basic; it's more complicated than that.

Or, consider Piet, an esolang in which programs are expressed as two-dimensional pixel paintings. Does a pixel count as a "token"? Does a rectangle of pixels count as a "token"?

In the same vein — of trying to break away from the strictures of linear words and into more free-flowing worlds — consider noit o' mnain worb, a nondeterministic cellular automaton. It uses ASCII characters to represent cells in the automaton; but do those characters each count as a "token", or are the relevant building blocks of worb programs more like "corridors" and "gates"?

For that matter, consider Wireworld, a better-known cellular automaton. Does Wireworld count as a "language"? and if so, does "it" "have" "tokens"?

For that matter, consider the "language" in which ENIAC was programmed. It certainly didn't have "tokens"! But if you were to write down the plugboard configuration for a given "program", I suppose that you'd pick some way of writing it down that involved ASCII and a grammar of tokens. I don't guess you'd communicate the plugboard configuration with a pen-and-ink drawing. (But maybe you would! Who knows!)

For that matter, what would you say about plain old x86 machine code? That's a "computer language" if ever there was one... but it doesn't have "tokens", does it? All it has is a stream of bytes, one after the other, which are chunked together to form instructions. Or does an instruction (such as B82A0000 or C3) count as a "token"? Does it matter if the same set of bytes (or some of them, at least) could be reinterpreted by another part of the program as a piece of data, a jump address, et cetera?

Basically, you need to define what you mean by "language" and "token"; and once you do that, I think you'll find that your question answers itself.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I can't imagine a language without tokens, but that could mean that my imagination is limited by my experience. The first platypus brought back to Europe (dead) was considered a fake because experience said such an animal could not exist. Atoms were originally thought to be indivisible. I don't see anything wrong with asking if languages must have tokens. If I use my own limited definitions of "language" and "token" the answer is no, but what if my assumptions are wrong? $\endgroup$
    – CJ Dennis
    Commented Sep 20, 2017 at 21:02
  • $\begingroup$ @CJDennis: There's a fundamental philosophical difference between those historical examples and your question. In the historical examples, everyone agreed on the meanings of the terms "animal", "atom", "indivisible", etc.; they merely held various conjectures about meaningful propositions such as "The platypus is an animal", "The atom is divisible", etc. In your question, nobody knows what you mean by the utterances "language" and "token". Without widespread agreement on the meaning of the words you're stringing together, your string of words is, by definition, meaningless. $\endgroup$
    – Quuxplusone
    Commented Sep 20, 2017 at 21:10
  • $\begingroup$ A better historical analogy might be from Chomsky: Can there be a colorless green idea that sleeps furiously? If you use your own limited definitions of "colorless", "green", "idea", and so on, then the answer is no; but what if your assumptions are wrong? $\endgroup$
    – Quuxplusone
    Commented Sep 20, 2017 at 21:14
  • $\begingroup$ Everyone agreed on the definition of "animal" and agreed it looked like an animal but that no single animal could have all those features, therefore it was a fake animal. $\endgroup$
    – CJ Dennis
    Commented Sep 20, 2017 at 21:15
  • $\begingroup$ If I'm like Humpty Dumpty and provide my own definition of words, I never need to use Stack Exchange because I'm satisfied with my own reasoning. $\endgroup$
    – CJ Dennis
    Commented Sep 20, 2017 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.