Why are so many internet protocols text-based?

Question

From what I have found, a very large amount of protocols that travel over the internet are "text-based" rather than binary. The protocols in question include, but are not limited to HTTP, SMTP, FTP (I think this one is all text-based?), WHOIS, IRC.

In fact, some of these protocols jump through some hoops whenever they want to transmit binary data.

Is there a reason behind this? Text-based protocols obviously have a bit of an overhead as they require sending more data to transmit the same amount of information (see example below). What benefits outweigh this?

---

By text-based, I mean most of the characters used in the protocol are between 0x20 (space) and 0x7E (~), with the occasional "special character" used for very special purposes, such as the newlines, null, ETX, and EOT. This is opposed to transmitting raw, binary data over the connection.

For instance, transmitting the integer 123456 as text would involve sending the string 123456 (represented in hex as 31 32 33 34 35 36), whereas the 32-bit binary value would be sent as (represented in hex) 0x0001E240 (and as you can see, "contains" the special null character.

Answer 1

When the world was younger, and computers weren't all glorified PCs, word sizes varied (a DEC 2020 we had around here had 36 bit words), format of binary data was a contentious issue (big endian vs little endian, and even weirder orders of bits were reasonably common). There was little consensus on character size/encoding (ASCII, EBCDIC were the main contenders, our DEC had 5/6/7/8 bits/character encodings). ARPAnet (the Internet predecessor) was designed to connect machines of any description. The common denominator was (and still is) text. You could be reasonably certain that 7-bit encoded text wouldn't get mangled by the underlying means to ship data around (until quite recently, sending email in some 8-bit encoding carried a guarantee that the recipient would get mutilated messages, serial lines were normally configured as 7-bit with one bit parity).

If you rummage around in e.g. the telnet or FTP protocol descriptions (the first Internet protocols, the network idea then was to connect remotely to a "supercomputer", and shuffle files to and fro), you see that the connection includes negotiating lots of details we take as uniform,

Yes, binary would be (a bit) more efficient. But machines and memories (and also networks) have grown enormously, so the bit scrimping of yore is a thing of the past (mostly). And nobody in their right mind will suggest ripping out all existing protocols to replace them with binary ones. Besides, text protocols offer a very useful debugging technique. Today I never install the telnet server (better use the encrypted SSH protocol for remote connections), but have to telnet client handy to "talk" to some errant server to figure out snags. Today you'd probably use netcat or ncat for futzing around...

Answer 2

One advantage that might be overlooked is the ability to experiment. If you're shoving bits down the tube, you're going to need to write some utility that translates EHLO into 0x18 or the like. Instead of doing that, you can simply telnet into a mail server, send EHLO and be on your way.

Nothing is preventing you in this day and age from writing code in Assembly or Brainf*ck, and you might very well save some bits by doing so. However, explaining what exactly you've done to someone else so they can understand and interact with your code is not going to be easy if you do so.

With protocols, it's important that users would be readily able to learn how to use them, as most people back in the day which were using ARPAnet or the beginnings of the Internet were people that felt comfortable behind a terminal.

Similar arguments, by the way, are held in companies today. Should we serialize to JSON or BSON (binary representation of JSON)? If you serialize to BSON, you shed some overhead, but you now need a translator to convert your BSON into JSON and vice-versa, as a human is going to have to read that data at some point when something inevitably goes wrong.

Answer 3

It's not that many internet protocols are text based. In fact, if I were to guess I'd say that text based protocols are in the minority. For almost every text based protocol you see on the internet there are at least two binary protocols that people have invented to send the same or similar data.

But it's true that the majority of internet traffic use text based protocols. This fact is interesting if you assume that there are many more binary protocols than text but many more text traffic than binary. It means that most of the successful protocols on the internet are text based. Except for a small number of applications (bittorrent is one example) binary protocols tend to die.

In the early days of the internet, corporations tended to design and use binary protocol (MSN for example, not the MSN website of today, the original proprietary MicroSoft Network that was supposed to replace HTTP) while the military, research institutes and academics tended to design and use text based protocol. Part of the reason was that building and debugging binary protocols was hard and corporations can afford to pay people to do it while the military, researchers and academics were doing it in their spare time for no pay (most of the people who developed the internet had jobs not related to developing the internet).

When you're writing code on weekends as a hobby and are not paid for doing what you do you tend to chose the simpler solution - text. So text based protocols got used by more people than binary protocols.

But that's not the full story. Building a network is hard. Really hard. We're so used to the internet today that we don't fully realize what a miracle of engineering it is. Almost every aspect of the internet evolved out of a bug fix. For example, we use IP address instead of MAC address because it allows us to build routers with just kilobytes (or these days megabytes) instead of terabytes of RAM for the routing table. The more and more problems we tried to solve, the more we tend to prefer text based protocols to debug them. Once we had enough experience developing the low-level network protocols, when it came time to develop application protocols most of the experienced programmers and engineers tended to prefer text protocols.

From personal experience, I've worked for a company building routers and I've also worked for a company building telemetry equipment so I've got a lot of experience working with binary protocols such as TCP/IP, ARP, IEC60870-5-101 and DNP3. I've also worked with text protocols such as HTTP, POP3 and NMEA. I've also worked with binary data formats such as ASN.1 and text data formats such as JSON and XML. If I were to choose I'd choose text almost every time. The only time I'd choose binary is if the protocol is really low-level (then I'd implement just enough so that I can plonk a text based protocol on top or it) or the data is naturally binary (like audio files).

Answer 4

Structured binary also has limitations in expanding it. It my days of working with FidoNet and building a gateway between it and UUCP/USNET, Fidonet's message headers were a structured binary. Expanding it by even just trying to add a byte someplace means breaking everything out there that is trying to work with it. Having a text header or protocol means you can expand something w/o breaking things.

Answer 5

Your question can be interpreted in three ways:

1. Why is numerical data transmitted in textual representation, as if it had been printed with e.g. printf()?
2. Why do the classical application layer protocols — e.g. the ftp control channel, smtp, http — traditionally all use a 7-bit ASCII character set? (7 bit ASCII can be considered "text" because most bytes correspond to printable glyphs or text control codes like newline and from feed.)
3. Why are blobs of binary data often converted to 7 bit ascii when they are sent over the internet, e.g. as a mail attachment?

The answer to the first one is interoperability. Integers and floating point values have different binary representations on different machines, or even compilers, or even with just different compiler options. Transmitting them effectively via printf/scanf makes interoperability easy. Note that this choice was only made for the higher level protocols of which a few are mentioned above; on the network layer data is transmitted binarily. For this, TCP/IP defines a binary integer representation, and libraries implementing TCP/IP provide means to convert between host and network representations with htonl and friends.

The answer to the second question is probably that RFC 206 (note the low number — 1971!) describes the telnet protocol, on which many application layer protocols are based, as a direct teletype replacement

whose function is to make an Online System terminal appear to any teletype-compatible, time-sharing system in the Network as if it were directly connected to that system.

(Emphasis in the original text.) At least some teletypes and in particular teletype networks used 7 bit ASCII as the character set which must have made it a natural choice.

The answer to the third one is simply that because the application layer protocols are telnet based, and telnet is 7 bit ascii, much soft- and hardware was not prepared to deal with 8 bit data. Sending binary attachments could be considered a misuse of email; hence the hoops. Today that's usually not true any longer and the protocols are continuously extended (or simply used) to directly handle binary data.

Answer 6

Another reason for this is that textual protocols are easier to interoperate with, and a lot more flexible for the future. Older generations, especially the folks behind Unix, already went through the dilemma between binary and textual. They ended up preferring textual for good reasons. It became, in fact, a programming best practice for Unix.

https://www.arp242.net/the-art-of-unix-programming/#ch05s01

Pipes and sockets will pass binary data as well as text. But there are good reasons the examples we'll see in Chapter 7 are textual: reasons that hark back to Doug McIlroy's advice quoted in Chapter 1. Text streams are a valuable universal format because they're easy for human beings to read, write, and edit without specialized tools. These formats are (or can be designed to be) transparent.

...

Designing a textual protocol tends to future-proof your system. One specific reason is that ranges on numeric fields aren't implied by the format itself. Binary formats usually specify the number of bits allocated to a given value, and extending them is difficult. For example, IPv4 only allows 32 bits for an address. To extend address size to 128 bits (as done by IPv6) requires a major revamping.[51] In contrast, if you need a larger value in a text format, just write it. It may be that a given program can't receive values in that range, but it's usually easier to modify the program than to modify all the data stored in that format. ...

There are exceptions where binary makes a lot more sense.

Just a recent example: I needed to collect Vault audit logs and filter some of them to send alerts to an email address, and send all of them to a remote sys log server.

Vault can send audit logs via TCP in JSON format separated with new lines, which is plain text. Fluentd can open a TCP port and receive JSON format elements separated by new lines, again, plain text. The email server SMTP protocol is, guess what, plain text. The Remote sys log server speaks plain text following RFC5424 formatting, on a TCP or UDP port.

I was able to hook them all to do the job, and all those components are from different companies and ages. Also, while working on the interoperability, I was able to fake some of the senders and receivers with a simple TCP python program.