0
$\begingroup$

Why does the BOM consist of two bytes instead of one for example in encoding utf-16? Information about the order of bytes can be easily saved on just one byte. Is it because it would be good for the sum of coded bytes to be even?

$\endgroup$

4 Answers 4

6
$\begingroup$

The BOM started out as an encoding trick. It was not part of Unicode, it was something that people discovered and cleverly (ab)used.

Basically, they found the U+FEFF Zero width non-breaking space character. Now, what does a space character that has a width of zero and does not induce a linebreak do at the very beginning of a document? Well, absolutely nothing! Adding a U+FEFF ZWNBSP to the beginning of your document will not change anything about how that document is rendered.

And they also found that the code point U+FFFE (which you would decode this as, if you decoded UTF-16 "the wrong way round") was not assigned. (U+FFFE0000, which is what you would get from reading UTF-32 the wrong way round, is simply illegal. Codepoints can only be maximal 21 bits long.)

So, what this means is that when you add U+FEFF to the beginning of your UTF-16 (or UTF-32) encoded document, then:

  • If you read it back with the correct Byte Order, it does nothing.
  • If you read it back with the wrong Byte Order, it is a non-existing character (or not a code point at all).

Therefore, it allows you to add a code point to the beginning of the document to detect the Byte Order in a way that works 100% of the time and does not alter the meaning of your document. It is also 100% backwards-compatible. In fact, it is more than backwards-compatible: it actually works as designed even with software that doesn't even know about this trick!

It was only later, after this trick had been widely used for many years that the Unicode Consortium made it official, in three ways:

  • They explicitly specified that the code point U+FEFF as the first code point of a document is a Byte Order Mark. When not at the beginning of the document, it is still a ZWNBSP.
  • They explicitly specified that U+FFFE will never be assigned, and is reserved for Byte Order Detection.
  • They deprecated U+FEFF ZWNBSP in favor of U+2060 Word Joiner. New documents that have an U+FEFF somewhere in the document other than as the first code point should no longer be created. U+2060 should be used instead.

So, the reason why the Byte Order Mark is a valid Unicode character and not some kind of special flag, is that it was introduced with maximum backwards-compatibility as a clever hack: adding a BOM to a document will never change it, and you don't need to do anything to add BOM detection to existing software: if it can open the document at all, the byte order is correct, and if the byte order is incorrect, it is guaranteed to fail.

If, on the other hand, you try to add a special one-octet signature, then all UTF-16 and UTF-32 reader software in the entire world has to be updated to recognize and process this signature. Because if the software does not know about this signature, it will simply try to decode the signature as the first octet of the first code point, and the first octet of the first code point as the second octet of the first code point, and further decode the entire document shifted by one octet. In other words: adding the BOM would completely destroy any document, unless every single piece of software in the entire world that deals with Unicode is updated before the first document with a BOM gets produced.

However, going back to the very beginning, and to your original question:

Why does the BOM consist of two bytes

It seems that you have a fundamental misunderstanding here: the BOM does not consist of two bytes. It consists of one character.

It's just that in UTF-16, each code point gets encoded as two octets. (To be fully precise: a byte does not have to be 8 bits wide, so we should talk about octets here, not bytes.) Note that in UTF-32, for example, the BOM is not 2 octets but 4 (0000FEFF or FFFE0000), again, because that's just how code points are encoded in UTF-32.

Note that the BOM is sometimes used even in cases where it is not necessary: for example, UTF-8 has no problems with byte order, since it fully defines it as part of the specification. The same applies to some more obscure formats such as UTF-1, UTF-5, UTF-6, UTF-7, UTF-7,5, UTF-EBCDIC, and so on. Here, a BOM is still sometimes used as a Unicode signature, i.e. to signal to a text processor that the document is encoded using a particular transfer encoding.

However, the use as a Unicode signature is generally discouraged, as is use of a BOM in cases where byte order can be specified by metadata (e.g. the charset parameter in HTTP), is fixed by some specification (e.g. filenames in Windows are always UTF-16LE, no BOM necessary), or can be deduced from the document (JSON does not need a BOM and in fact does not need a charset parameter either, you can always tell by reading the first 4 octets whether it is UTF-8, UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE; actually, BOMs are forbidden and charset parameters are discouraged and useless).

$\endgroup$
3
  • 2
    $\begingroup$ "It seems that you have a fundamental misunderstanding here: the BOM does not consist of two bytes. It consists of one character." That's the essential problem. And note that BOMs in UTF-8 and UTF-32 use the same character (or code point more precise). $\endgroup$
    – gnasher729
    Commented Sep 9, 2021 at 22:03
  • $\begingroup$ Great answer. It would be even better if you mentioned the UTF-8 case. $\endgroup$
    – Nayuki
    Commented Sep 10, 2021 at 3:50
  • 2
    $\begingroup$ It's not "100% backwards-compatible". A lot of text files are designed to be read by machine and have significant whitespace, or the lexer may just not recognize ZWNBSP as whitespace. Unaware tools like cat can migrate ZWNBSPs to the middle of files, etc. $\endgroup$
    – benrg
    Commented Sep 10, 2021 at 4:34
2
$\begingroup$

There is no guarantee that a file has a Byte Order Marker. For example, JSON documents don't need one at all, no matter what the encoding is (as a little puzzle, figure out how you can determine which Unicode encoding a JSON document is using).

Now if you KNOW there is a single byte BOM, that's easy. But what do you do if you don't know? If the first character in your file just matches the single byte BOM by chance? You're stuck.

Another problem is that you couldn't store data with a single byte BOM into an array of 16 bit integers. That's a bit awful.

$\endgroup$
2
$\begingroup$

In addition to what Jörg W Mittag said:

The Unicode Consortium isn't in the business of defining file formats. I think the original standard didn't even define any encoding of Unicode as a sequence of bytes – that was an implementation detail.

In a sane world, there wouldn't be any need for something like the BOM, because the format of the character data would be determined by metadata provided by the sender and understood by the receiver. But nothing in the software world ever works. Half of the time there's no metadata, and half of the time that there is metadata, it's wrong.

The BOM is the result of the Unicode Consortium trying to deal with a real-world problem while clinging to some semblance of the idea that they're just standardizing a character encoding and not every aspect of text processing.

$\endgroup$
1
$\begingroup$

In UTF-16, information is encoded in units of 2 bytes. Every symbol is encoded using either one unit or two units. BOM uses one unit.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.