47
$\begingroup$

From what I have found, a very large amount of protocols that travel over the internet are "text-based" rather than binary. The protocols in question include, but are not limited to HTTP, SMTP, FTP (I think this one is all text-based?), WHOIS, IRC.

In fact, some of these protocols jump through some hoops whenever they want to transmit binary data.

Is there a reason behind this? Text-based protocols obviously have a bit of an overhead as they require sending more data to transmit the same amount of information (see example below). What benefits outweigh this?


By text-based, I mean most of the characters used in the protocol are between 0x20 (space) and 0x7E (~), with the occasional "speical character" used for very special purposes, such as the newlines, null, ETX, and EOT. This is opposed to transmitting raw, binary data over the connection.

For instance, transmitting the integer 123456 as text would involve sending the string 123456 (represented in hex as 31 32 33 34 35 36), whereas the 32-bit binary value would be sent as (represented in hex) 0x0001E240 (and as you can see, "contains" the special null character.

$\endgroup$
  • 3
    $\begingroup$ Out of 5 mentioned protocols, HTTP, SMTP, WHOIS and IRC were primarily conceived to exchange textual data. $\endgroup$ – el.pescado Sep 25 '15 at 21:01
  • 4
    $\begingroup$ Note that HTTP/2 is a binary protocol. $\endgroup$ – isanae Sep 25 '15 at 21:16
  • 4
    $\begingroup$ You're mostly referring to application and presentation layer protocols. Lower level protocols (TCP, IP, Ethernet) are almost always binary. $\endgroup$ – Nick T Sep 26 '15 at 3:27
  • 2
    $\begingroup$ FTP has a binary mode which was quite important to use when transferring binary files, as the normal transfer mode in many clients would re-write line endings to match the host convention which would corrupt binaries when transferring between hosts with different line endings. This binary mode was only for file transfer and didn't effect the command stuff. $\endgroup$ – casey Sep 27 '15 at 19:20
  • 1
    $\begingroup$ FTP actually uses two network connections, one text-based (the command channel) and one binary (the data channel). $\endgroup$ – Pseudonym Sep 28 '15 at 1:14
39
$\begingroup$

When the world was younger, and computers weren't all glorified PCs, word sizes varied (a DEC 2020 we had around here had 36 bit words), format of binary data was a contentious issue (big endian vs little endian, and even weirder orders of bits were reasonably common). There was little consensus on character size/encoding (ASCII, EBCDIC were the main contenders, our DEC had 5/6/7/8 bits/character encodings). ARPAnet (the Internet predecessor) was designed to connect machines of any description. The common denominator was (and still is) text. You could be reasonably certain that 7-bit encoded text wouldn't get mangled by the underlying means to ship data around (until quite recently, sending email in some 8-bit encoding carried a guarantee that the recipient would get mutilated messages, serial lines were normally configured as 7-bit with one bit parity).

If you rummage around in e.g. the telnet or FTP protocol descriptions (the first Internet protocols, the network idea then was to connect remotely to a "supercomputer", and shuffle files to and fro), you see that the connection includes negotiating lots of details we take as uniform,

Yes, binary would be (a bit) more efficient. But machines and memories (and also networks) have grown enormously, so the bit scrimping of yore is a thing of the past (mostly). And nobody in their right mind will suggest ripping out all existing protocols to replace them with binary ones. Besides, text protocols offer a very useful debugging technique. Today I never install the telnet server (better use the encrypted SSH protocol for remote connections), but have to telnet client handy to "talk" to some errant server to figure out snags. Today you'd probably use netcat or ncat for futzing around...

$\endgroup$
  • 10
    $\begingroup$ Ease of troubleshooting is greatly improved as well. Reading a packet capture is difficult enough, it's made even worse when applications don't send messages in human-readable format. $\endgroup$ – Nanban Jim Sep 25 '15 at 18:59
  • 5
    $\begingroup$ "And nobody in their right mind will suggest ripping out all existing protocols to replace them with binary ones" -- rather, you negotiate your way up from the text-based protocols to the thing you think is better, as from HTTP to what was SPDY request header compression and is now part of HTTP/2. Or, for that matter, from HTTP to binary content-types or transfer-encodings. $\endgroup$ – Steve Jessop Sep 25 '15 at 19:05
  • 4
    $\begingroup$ Plain-text protocols also lets you safely examine potentially dangerous or untrusted data. For example, I use telnet when I receive some spam/phishing attempt, which I can virtually guarantee won't harm my system. Having text-based access to a system is critical. Even today, though, you'll note that HTTP/1.1 is rarely "plain text", because the Accept-Encoding header allows for compression, which most browsers user and servers support, in order to load pages faster. $\endgroup$ – phyrfox Sep 25 '15 at 20:19
  • $\begingroup$ At the Vintage Computer Fair of the Midwest, I found it interesting that machines like the Altair 680 needed to receive code in Motorola S-record format, which used 76 characters for every 32 bytes of data (44 characters of overhead). Even if one were limited to using a 41-character set like 0-9 A-Z +-*/= it should still be possible to reduce that to something closer to 57 characters (25 characters of overhead), which would reduce the time for an ASR-33 to feed 1K of code from 4 minutes to about three. Given the slow I/O speeds, I wonder why such things don't seem to have been commonly done? $\endgroup$ – supercat Sep 25 '15 at 20:33
24
$\begingroup$

One advantage that might be overlooked is the ability to experiment. If you're shoving bits down the tube, you're going to need to write some utility that translates EHLO into 0x18 or the like. Instead of doing that, you can simply telnet into a mail server, send EHLO and be on your way.

Nothing is preventing you in this day and age from writing code in Assembly or Brainf*ck, and you might very well save some bits by doing so. However, explaining what exactly you've done to someone else so they can understand and interact with your code is not going to be easy if you do so.

With protocols, it's important that users would be readily able to learn how to use them, as most people back in the day which were using ARPAnet or the beginnings of the Internet were people that felt comfortable behind a terminal.

Similar arguments, by the way, are held in companies today. Should we serialize to JSON or BSON (binary representation of JSON)? If you serialize to BSON, you shed some overhead, but you now need a translator to convert your BSON into JSON and vice-versa, as a human is going to have to read that data at some point when something inevitably goes wrong.

$\endgroup$
  • $\begingroup$ If protocols had been designed as binary in the first place, rather than a binary shorthand for a text protocol, there might not even be a commonly-agreed-upon term like EHLO. Each human-usable frontend for the binary protocol might have made up their own name, if the binary standard didn't name 0x18-in-this-position. $\endgroup$ – Peter Cordes Sep 27 '15 at 14:24
10
$\begingroup$

It's not that many internet protocols are text based. In fact, if I were to guess I'd say that text based protocols are in the minority. For almost every text based protocol you see on the internet there are at least two binary protocols that people have invented to send the same or similar data.

But it's true that the majority of internet traffic use text based protocols. This fact is interesting if you assume that there are many more binary protocols than text but many more text traffic than binary. It means that most of the successful protocols on the internet are text based. Except for a small number of applications (bittorrent is one example) binary protocols tend to die.

In the early days of the internet, corporations tended to design and use binary protocol (MSN for example, not the MSN website of today, the original proprietary MicroSoft Network that was supposed to replace HTTP) while the military, research institutes and academics tended to design and use text based protocol. Part of the reason was that building and debugging binary protocols was hard and corporations can afford to pay people to do it while the military, researchers and academics were doing it in their spare time for no pay (most of the people who developed the internet had jobs not related to developing the internet).

When you're writing code on weekends as a hobby and are not paid for doing what you do you tend to chose the simpler solution - text. So text based protocols got used by more people than binary protocols.

But that's not the full story. Building a network is hard. Really hard. We're so used to the internet today that we don't fully realize what a miracle of engineering it is. Almost every aspect of the internet evolved out of a bug fix. For example, we use IP address instead of MAC address because it allows us to build routers with just kilobytes (or these days megabytes) instead of terabytes of RAM for the routing table. The more and more problems we tried to solve, the more we tend to prefer text based protocols to debug them. Once we had enough experience developing the low-level network protocols, when it came time to develop application protocols most of the experienced programmers and engineers tended to prefer text protocols.

From personal experience, I've worked for a company building routers and I've also worked for a company building telemetry equipment so I've got a lot of experience working with binary protocols such as TCP/IP, ARP, IEC60870-5-101 and DNP3. I've also worked with text protocols such as HTTP, POP3 and NMEA. I've also worked with binary data formats such as ASN.1 and text data formats such as JSON and XML. If I were to choose I'd choose text almost every time. The only time I'd choose binary is if the protocol is really low-level (then I'd implement just enough so that I can plonk a text based protocol on top or it) or the data is naturally binary (like audio files).

$\endgroup$
3
$\begingroup$

Structured binary also has limitations in expanding it. It my days of working with FidoNet and building a gateway between it and UUCP/USNET, Fidonet's message headers were a structured binary. Expanding it by even just trying to add a byte someplace means breaking everything out there that is trying to work with it. Having a text header or protocol means you can expand something w/o breaking things.

$\endgroup$
  • $\begingroup$ Lesson learned: Put a version tag in binary data. $\endgroup$ – Peter A. Schneider Jul 12 '18 at 8:59
3
$\begingroup$

Your question can be interpreted in three ways:

  1. Why is numerical data transmitted in textual representation, as if it had been printed with e.g. printf()?
  2. Why do the classical application layer protocols — e.g. the ftp control channel, smtp, http — traditionally all use a 7-bit ASCII character set? (7 bit ASCII can be considered "text" because most bytes correspond to printable glyphs or text control codes like newline and from feed.)
  3. Why are blobs of binary data often converted to 7 bit ascii when they are sent over the internet, e.g. as a mail attachment?

The answer to the first one is interoperability. Integers and floating point values have different binary representations on different machines, or even compilers, or even with just different compiler options. Transmitting them effectively via printf/scanf makes interoperability easy. Note that this choice was only made for the higher level protocols of which a few are mentioned above; on the network layer data is transmitted binarily. For this, TCP/IP defines a binary integer representation, and libraries implementing TCP/IP provide means to convert between host and network representations with htonl and friends.

The answer to the second question is probably that RFC 206 (note the low number — 1971!) describes the telnet protocol, on which many application layer protocols are based, as a direct teletype replacement

whose function is to make an Online System terminal appear to any teletype-compatible, time-sharing system in the Network as if it were directly connected to that system.

(Emphasis in the original text.) At least some teletypes and in particular teletype networks used 7 bit ASCII as the character set which must have made it a natural choice.

The answer to the third one is simply that because the application layer protocols are telnet based, and telnet is 7 bit ascii, much soft- and hardware was not prepared to deal with 8 bit data. Sending binary attachments could be considered a misuse of email; hence the hoops. Today that's usually not true any longer and the protocols are continuously extended (or simply used) to directly handle binary data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.