1
$\begingroup$

Here is a code point: U+091D. The symbol it represents in UTF-8 is झ. In hex the symbol requires three bytes: e0 a4 9d. But it looks like the number 091D requires two bytes. So why do we need three bytes to encode the symbol? Probably due to

the restriction of the Unicode code-space to 21-bit values in 2003

Please see the beginning of the description section. OK, but why did they restrict the code-space to 21 bits?

I converted e0 a4 9d to binary, and the result is

11100000 10100100 10011101

Then I converted 091D to binary and got

00001001 00011101

The two binary results seems to have little in common. So how did 091D become e0 a4 9d?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ By the way, there no such a thing as a "UTF-8 code point". You can think of Unicode code points as "numbers" associated to characters. This association does NOT specify how to store such numbers in a file -- this depends on the encoding. Unicode suggests several encodings to represent code points: utf7, utf8, utf16 (le/be), ... (even if, IMO, only utf8 should be used, in files). It is important to keep separate in one's mind the data (the code point) and the encoding (the sequence of bytes). $\endgroup$ – chi Jun 7 '17 at 12:04
  • $\begingroup$ @chi, you.re right. Here is the definition of code point: unicode.org/glossary/#code_point. I will remove the word UTF-8 there. $\endgroup$ – Maksim Dmitriev Jun 8 '17 at 6:03
5
$\begingroup$

Where the 21 bits come from: The idea of unicode is based on the Universal Coded Character Sets (short UCS). It's a concept for a 31bit character set ordered as a 4D hypercube where the first three dimensions use 8bit and the fourth uses 7bit. Per row there are $2^8$ characters. Per plane there are $2^8$ rows $=65.536$ characters. Per cube there are $2^8$ planes and there are $2^7$ cubes. Unicode decided to just use a mere 17 planes (the code space 0 - 10FFFF), which corresponds to the 21bits.

UTF-8 is a variable length encoding mostly used for encoding unicode. Variable length means that it uses 1 to 4 byte to represent a certain code point, depending on its number of significant bits.

The scheme looks as following:

1 byte: At most 7 significant bits. From U+0000 to U+007F.
Scheme: 0xxxxxxx.

2 bytes: At most 11 significant bits. From U+0080 to U+07FF.
Scheme: 110xxxxx 10xxxxxx

3 bytes: At most 16 significant bits. From U+0800 to U+FFFF.
Scheme: 1110xxxx 10xxxxxx 10xxxxxx

4 bytes: At most 21 significant bits. From U+10000 to U+10FFFF.
Scheme: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

In your example you have 12 significant bits and therefore the 3 bytes used for encoding.

For further details see https://en.wikipedia.org/wiki/UTF-8.

Improve this answer
$\endgroup$
6
  • $\begingroup$ In your answer I didn't find an answer to my question: why did they restrict the code-space to 21 bits. $\endgroup$ – Maksim Dmitriev Jun 7 '17 at 5:17
  • 1
    $\begingroup$ I changed the answer accordingly. $\endgroup$ – Burgberg Jun 7 '17 at 11:23
  • $\begingroup$ Interesting explanation! I also found one on en.wikipedia.org/wiki/UTF-8: The leading bytes and the continuation bytes do not share values (continuation bytes start with 10 while single bytes start with 0 and longer lead bytes start with 11). This means a search will not accidentally find the sequence for one character starting in the middle of another character. It also means the start of a character can be found from a random position by backing up at most 3 bytes to find the leading byte. $\endgroup$ – Maksim Dmitriev Jun 8 '17 at 6:55
  • $\begingroup$ That's exactly how it works. $\endgroup$ – Burgberg Jun 8 '17 at 9:59
  • 1
    $\begingroup$ @MaksimDmitriev: bits, not bytes. $\endgroup$ – gnasher729 Jun 8 '17 at 19:11
0
$\begingroup$

The code space isn't just restricted to 21 bits, it's restricted to values from 0 to 0x10ffff. 21 bits would be 0 to 0x1fffff, almost twice as many values.

The reason for the restriction is that this is intended for the storage of code points representing characters, and the Unicode standard doen't think that more than 17 x 65536 code points are or will ever be required. It also just so happens that the range from 0 to 0x10ffff is what can be easily represented in UTF-16 with up to two values, so a bigger range would cause problems there.

You could use a very similar scheme with up to 7 bytes storing integers up to 36 bit, but Unicode doesn't use that scheme.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.