44
$\begingroup$

It seems to me that Unicode is the "final" character encoding. I cannot imagine anything else replacing it at this point. I'm frankly confused about why UTF-16 and UTF-32 etc. exist at all, not to mention all the non-Unicode character encodings (unless for legacy purposes).

In my system, I've hardcoded UTF-8 as the one and only supported character encoding for my database, my source code files, and any data I create or import to my system. My system internally works solely in UTF-8. I cannot imagine ever needing to change this, for any reason.

Is there a reason I should expect this to change at some point? Will UTF-8 ever become "obsolete" and replaced by "UniversalCode-128" or something, which also includes the alphabets of later discovered nearby galaxies' civilizations?

Improve this question
$\endgroup$
19
  • 8
    $\begingroup$ "I've hardcoded UTF-8": I hope that you meant explicit hardcoding by that. Many, if not all, encoding problems stem from implicit hardcodings and wrong assumptions about them from external software. $\endgroup$ – ComFreek Jun 14 '20 at 16:15
  • 12
    $\begingroup$ "I'm frankly confused about why UTF-16 and UTF-32 etc. exist at all". The first version of Unicode was restricted to 16-bit codepoints, which should explain why UTF-16 was an obvious choice back then. Backwards compatibility explains why it's still to this day. $\endgroup$ – Voo Jun 14 '20 at 17:32
  • 15
    $\begingroup$ Just keep in mind, that UTF-16 is now also variable-length, @OrangeDog ;-) $\endgroup$ – René Jun 14 '20 at 20:14
  • 2
    $\begingroup$ @René indeed (it always was, UCS2 is the fixed one), though still more efficient to check than UTF-8, or you can decide not to support the higher planes. $\endgroup$ – OrangeDog Jun 14 '20 at 21:44
  • 3
    $\begingroup$ @hanshenrik But you never want to deal with codepoints but graphemes (the nth codepoint might not even be valid on its own). $\endgroup$ – Voo Jun 16 '20 at 7:58
33
$\begingroup$

UTF-8 might not last forever, but you probably don't have to worry too much.

Two universal truths:

  • We can't predict the future.
  • Nothing lasts forever, especially in software.

But that doesn't mean the benefit of (trying to) future-proof your code always outweighs the cost.

Is UTF-8 likely to become obsolete any time soon?

I would say no. UTF-8 is quite common, which makes it harder to replace. Unicode also still has quite a bit of empty space, meaning there isn't likely to be a pressing need to replace it soon. Between 2010 and 2020, less than 40k characters have been added. It will take about 240 years to use up the remaining ~1 million unallocated characters if we assume we keep going at the same rate. This is a lot faster than I imagined, but still quite a while away and assuming it will keep going at the same rate is quite an assumption.

It also doesn't seem like there'd be a need to replace it due to a fundamental flaw in the encoding. With other types of standards or technologies there may be some security issue that could be exploited, but this doesn't seem likely with character encodings that only tells you how characters are stored.

I speculate if a need to replace it arises, it would be due to inefficiencies or constraints in new technology. Someone could develop some new piece of technology that rethinks how data is stored or loaded, which might make UTF-8 less than ideal or unusable. But there would still be plenty of systems without that technology for quite a few years.

Note that I didn't ask "are we likely to see a new character encoding any time soon". Anyone can create a new standard, but that doesn't mean it will be widely adopted nor replace other standards.

How bad would it be for you if there's a new standard?

Probably not that bad.

Even if there is a new standard that's widely adopted, your system will likely keep functioning for the foreseeable future with little to no changes. There are a lot of legacy systems out there.

If your system doesn't support the new encoding, you may have some issues with the user or other systems trying to send you data you don't support. But your system could still use UTF-8 internally, even if this means you don't support some characters (which might not be good, but it won't necessarily break your system).

Also, if it were to be replaced due to a reason other than running out of space (which, as noted above, doesn't seem likely any time soon), UTF-8 could likely be extended to include any characters in the new encoding. Meaning you can just convert from one encoding to the other where required and UTF-8 would still be usable.

Unicode versus Unicode?

The difference between UTF-8, UTF-16 and UTF-32 seems minor when compared to other (non-Unicode) encodings. They all support the same characters, so it shouldn't be a huge issue if one replaces the other.

If another one of those were to become the widely adopted one, it would probably be trivial to convert between them where required and continue to use UTF-8 everywhere else.

Improve this answer
$\endgroup$
6
  • 10
    $\begingroup$ Nitpick/clarification: Unicode is the mapping between characters and numbers (and, well, a bunch of other related stuff necessary to reproduce written text). But Unicode itself doesn't tell anything about how to store thess numbers in binary bytes. That's where UTF-7/8/16/32 come into play. They don't say anything about which number means what, but they do specify how to convert numbers into bytes. Unicode itself doesn't have a maximum limit of characters, it's abstract. UTF does (but there's still plenty of space left). $\endgroup$ – Vilx- Jun 14 '20 at 23:47
  • 2
    $\begingroup$ @Vilx-: The number in the answer (about a million) is familiar. IIRC, it is intentional to ensure that UTF-8 encodings take at most 4 bytes, but it is a Unicode restriction. (16 "planes" of 64K characters each). UTF-8 has a trivial extension to 5 and 6 byte encodings. $\endgroup$ – MSalters Jun 15 '20 at 9:36
  • 1
    $\begingroup$ There's one possibly-security-affecting issue with UTF-8: overlong character encoding, which may be used to circumvent lists of unallowed strings. UTF-16 surrogate characters elegantly avoided this by adding an offset $\endgroup$ – Jonathan Jun 15 '20 at 10:35
  • 6
    $\begingroup$ @Vilx- nitpicking on your nitpick then: Unicode is a standard that defines both the database of characters and the encoding forms UTF-8, UTF-16 and UTF-32. So the Unicode standard explicitly explains how to store "these numbers" in bytes, and UTF-8/16/32 always encode Unicode characters. The standard also defines the limit to the number of characters: "Unicode codespace: A range of integers from 0 to 10FFFF" $\endgroup$ – Erwin Bolwidt Jun 15 '20 at 11:55
  • 1
    $\begingroup$ @MSalters If you mean that it is a UTF-16 restriction pushed up to Unicode itself (and thus the other UTFs), that's true enough. $\endgroup$ – Deduplicator Jun 16 '20 at 11:39
19
$\begingroup$

When it comes to software, the future always means needing to handle more data--- bigger files, and more of them in a shorter period of time. How does UTF-8 processing scale in those situations?

UTF-8 uses a variable number of bytes per character. This saves a lot of space if your text is ASCII plus the occasional emoji or accented letter. But a drawback of variable-length encoding is that jumping to an arbitrary position scales linearly with the size of the document. A fixed width encoding like UTF-32 would use more space but jumping to a position in the document is constant time. Depending on the size of the document and speed of the medium you're reading it from, linear time seeking vs. constant time seeking could make a huge difference in the performance of your application. It is better to be able to tradeoff space for time or the reverse as the situation demands.

Improve this answer
$\endgroup$
6
  • 19
    $\begingroup$ "jumping to a position in the document is constant time". Well jumping to the nth Unicode codepoint is constant time. Which is virtually useless (I can't think of a single use case where that's correct), because what we really care about are graphemes. UTF-32 requires linear time for that just as UTF-8 does, it just has to go through way more memory while doing so, therefore is slower. $\endgroup$ – Voo Jun 14 '20 at 17:20
  • 10
    $\begingroup$ You can bypass the need for linear seeking by indexing the file. If you mark down the position of every 8192th character in a file, you only increase the file size by 0.1% and get a very fast constant-time seek by character index. $\endgroup$ – John Dvorak Jun 14 '20 at 17:23
  • 5
    $\begingroup$ Jumping to a position in the document in UTF-8 is constant time if you store "position in the document" appropriately - there's no good reason (other than legacy) to store that position as the number of graphemes since the beginning of the document. For every interesting position that you need to store, you can store an appropriate "direct pointer" to the appropriate location, the scenario where your true underlying need is to jump to 12345th grapheme seems quite convoluted. $\endgroup$ – Peteris Jun 14 '20 at 23:50
  • 2
    $\begingroup$ On the other hand, there are also performance benefits - if you need to iterate over a string, then a dense variable-length representation will generally be faster than a sparse fixed-length representation since it's more "friendly" to cache lines and memory bandwidth - heck, in some situations it's faster to store zipped data in memory as simply streaming the full data from memory can be slower than streaming fewer bytes and spending some CPU cycles to decompress them. $\endgroup$ – Peteris Jun 14 '20 at 23:55
  • 6
    $\begingroup$ A key problem with the assertions in this answer is that there is not a 1:1 mapping between unicode code points and things a user would think of as a character. So if you want to go to "user perceived character n" then utf-32 doesn't help you. $\endgroup$ – Peter Green Jun 15 '20 at 1:36
9
$\begingroup$

UTF-8 might not last forever, but if you permit long UTF-8 again, it will outlast all other encodings that exist today. I have heard it projected that we will eventually run out of UTF-16 codepoints, necessitating the abandoning of UTF-16. We can go all the way to 0x7FFFFFFF.

Table from Wikipeida:

1   U+0000     U+007F     0xxxxxxx                  
2   U+0080     U+07FF     110xxxxx  10xxxxxx                
3   U+0800     U+FFFF     1110xxxx  10xxxxxx  10xxxxxx          
4   U+10000    U+1FFFFF   11110xxx  10xxxxxx  10xxxxxx  10xxxxxx        
5   U+200000   U+3FFFFFF  111110xx  10xxxxxx  10xxxxxx  10xxxxxx  10xxxxxx  
6   U+4000000  U+7FFFFFFF 1111110x  10xxxxxx  10xxxxxx  10xxxxxx  10xxxxxx  10xxxxxx

There's some debate how to extend this should it ever arise Do we go 1111111x for 0x7FFFFFFF to 0xFFFFFFFF and say larger code points cannot occur, or we do 11111110 beginning a 7 byte sequence for 36 bits of codepoint and permitting 11111111 for 8 byte sequence.

Editorial: I do not mind the fact this answer is mildly controversial. The whole answer is about the fact that UTF-8 and UTF-32 are more future-proofed than any other well-known encodings.

Improve this answer
$\endgroup$
6
  • $\begingroup$ What advantage do code points have over entities or grapheme clusters? If UTF-8 were to use a couple of the presently-unused 8-bit values as "start of entity" and "end of entity" codes conceptually similar to the way & and ; work in HTML, but used 10xxxxxx bytes instead of alphanumerics for the middle portion, such a design would be arbitrarily extensible to accommodate an unlimited number of glyphs without processing software having to know or care about what glyphs might be added in future. $\endgroup$ – supercat Jun 14 '20 at 19:33
  • 2
    $\begingroup$ @supercat: A radical new scheme is not likely to have better longevity than UTF-8. $\endgroup$ – Joshua Jun 14 '20 at 21:06
  • $\begingroup$ My point is that I don't see any need for over 1,000,000 code points, since there are already mechanisms for combining code points to yield an arbitrary variety of characters. At present, they're a bit bulky, but they could be represented more compactly. $\endgroup$ – supercat Jun 14 '20 at 21:19
  • 2
    $\begingroup$ Unicode itself is abstract and doesn't have any limitations on how big numbers it can use. However it also doesn't specify how to store the big numbers into binary bytes. UTF takes care of this technical detail and that does have limits. However when we run into them we can just invent another encoding scheme while leaving Unicode itself unchanged. $\endgroup$ – Vilx- Jun 14 '20 at 23:51
  • 2
    $\begingroup$ Re "Table from Wikipeida", That's very misleading! That table has nothing to do with Unicode of UTF-8. It's from earlier drafts of Unicode before they went with a smaller range. /// Re "There's some debate how to extend this should it ever arise", I doubt that. Unicode only goes up to U+10FFFF, so the table you posted already provides a way of extending Unicode by a factor of almost 2,000!!! $\endgroup$ – ikegami Jun 15 '20 at 6:09
9
$\begingroup$

UTF-8 is an elegant hack to remain backward compatible with ASCII and trivially compatible with Latin-1, which were both widely entrenched when Unicode started to take hold. UTF-8 can be extended further and still remain backward compatible with itself, by adding 5- and 6-byte encodings. So if Unicode decides it needs a few more bits to represent its character repertoire, there will be some hitches to make sure you programs get updated, but your existing data should be just fine. (Just as UTF-16 is backward compatible with UCS-2.) UTF-8 is deeply entrenched, so if it ever becomes obsolete, the new encoding system will almost certainly be backward compatible with UTF-8. Your existing data won't need to be converted, just as your ASCII documents are still perfectly good today.

How could UTF-8 become obsolete? It seems like Unicode has so much room for expansion that running out is almost unimaginable. On the other hand...

  • Unicode did run out of space once before, when it was a 16-bit system. I believe the limits of 16-bit was an impetus for the Han Unification, which combined substantially similar characters among several Pacific Asian languages.

  • Our idea of text being a serial stream of code points could evolve to the point that the Unicode approach isn't sufficient or scalable. Current best practice is to split content from styling almost completely. But it's hard to get that separation right (see CSS and evolving markup languages). It's not too hard to imagine at least some styling creeping back into the textual representation. Depending how that's done, it could have a massive multiplicative effect on the scale of Unicode.

    In fact, some of this has already happened. The Han Unification largely works, but to render a multi-lingual document properly, you need to know which spans of CJK symbols are Chinese, Japanese, or Korean. Because while the general shape of the unified symbols are the same and the concepts they represent align, they generally should be drawn with language-specific fonts. If you have just the text and not the styling, it's impossible for a machine to know which strings are from which language. So Unicode has a way to add language tags to get it right (just like you need some special characters to handle some Bidi edge cases). This is arguably styling (or, at least, mark-up) embedded directly in the text. And they're not widely supported.

  • Emoji. I was surprised that Unicode adopted emoji (beyond a handful to preserve legacy documents). In my mind, it doesn't seem to fit what Unicode set out to do, but the consortium's membership includes smart phone makers. There are many emoji, and the number is growing at quick pace. Combining characters are used to style your smiley by setting the character's gender, hair color, skin tone, occupation, etc. Emoji are becoming a generative script.

  • Icons. Now that we have a wider range of device resolutions and some extensions to font technology (thanks to emoji), software is turning to fonts for clean resolution-independent icon rendering. Unicode has recognized a couple hundred wing dings, so why not icons? If they can be assigned a semantic meaning (e.g., "SAVE ICON") instead of a descriptive name (e.g., "FLOPPY DISC ICON"), all the better. And if they start to fold in some styling information (e.g., "SAVE ICON", "DISABLED SAVE ICON", "PRESSED SAVE ICON", ...) we could see a massive number of these becoming standardized.

  • Private use. Currently, private use areas are used for icons (as above), for corporate logos, and sometimes even for original names (which I've heard is or was a trending in Japan). Documents with private use code points have semantic gaps and are inherently tied to styling information (custom fonts). I wouldn't be surprised if Unicode doesn't eventually start to allocate dedicated code points to corporate logos, and/or we'll see styling slither into our text documents.

  • Aliens. This won't happen for a long time, but it's easy to imagine alien languages being written in ways that cannot be represented as a linear stream of code points. What if the alien's script cannot be divorced from styling information? What if they have a generative writing system that cannot be reproduced with finite sets of glyphs, combining marks, and shaping rules?

Improve this answer
$\endgroup$
2
  • $\begingroup$ " It's not too hard to imagine at least some styling creeping back into the textual representation." It already has. Reading the sentence 'The Hebrew words for "one", "two" and "three" are "אחד", "שתיים", and "שלוש", respectively, would you guess that the Hebrew word for "one" is "אחד", and the word for "two" is "שתיים"? $\endgroup$ – supercat Jun 15 '20 at 21:03
  • $\begingroup$ There are already problems representing some human languages as linear streams of code points. Look up the wrangles about representing Sutton SignWriting in Unicode. $\endgroup$ – TRiG Jun 16 '20 at 11:01
6
$\begingroup$

Your question appears to slightly conflate two related concepts (as people often do):

  1. Unicode is a standard, whose primary part is a "coded character set" - a list of "code points", and a lot of metadata around them, attempting to catalogue all the world's writing systems. It has a defined "code space" of the numbers 0 to 10FFFF (hexadecimal) inclusive (most of which has not yet been filled with actual defined code points).
  2. UTF-8, UTF-16, and various other "encoding schemes" are ways of storing and transmitting Unicode code points. They can all represent all the code points, present and future, that the Unicode code space can theoretically hold.

When you talk about "something which also includes the alphabets of later discovered nearby galaxies' civilizations", you are implicitly talking about superseding or extending the Unicode character set itself.

If your system can "only" represent the code points of Unicode as we know it today (regardless of how it represents them), it would need to be upgraded if you needed to store these extra alphabets. It's impossible to say what this would involve; the new system might be cleverly designed to allow easy upgrades, or it might be that we adopt a system from Alpha Centauri, and all Unicode text needs carefully re-processing into their system. At that point, whether you picked UTF-8 or UTF-EBCDIC to store your Unicode would feel like an irrelevant detail.

If Unicode is not superseded or extended, any system capable of storing all Unicode code points will remain capable of storing them. So the theoretical limitations of UTF-8 in particular are not to do with what it can store, but how convenient it is to work with.

Currently, UTF-8 is the most popular encoding scheme, for various reasons - it has backwards-compatibility with ASCII, is compact when storing text containing mostly Latin characters, and works in multiples of 8 bits. Consequently, there are many tools for working with it - the virtuous cycle of standardisation. However, a new encoding scheme might become popular due to changes in common requirements - for instance, given an extremely "wide" memory, you could allocate a fixed width for each grapheme (i.e. even wider than the 32 bits required to fix the width of each code point).

If that happened, we can see what the upgrade would look like - you would need to convert your UTF-8 text to and from this encoding scheme in order to use tools built for it, which might be slow. But if you were still representing Unicode code points, such a transform is guaranteed to be possible without losing any data in either direction.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Beware of combinatorical explosion and virtually unlimited grapheme-size: Fixed-width seems a pipe-dream even if you go to ridiculous bitcounts. $\endgroup$ – Deduplicator Jun 16 '20 at 11:47
  • $\begingroup$ @Deduplicator Sure, but since the other given example was "civilizations from neighbouring galaxies", pipe dreams seemed reasonable. As an aside, though, Raku (Perl6) has a concept called "NFG" which puts multi-code-point graphemes into a lookup table; a serialization format based on that concept probably wouldn't be practical, but it's not impossible someone could make it work in some hypothetical distant future. $\endgroup$ – IMSoP Jun 16 '20 at 13:18
5
$\begingroup$

I'm frankly confused about why UTF-16 and UTF-32 etc. exist at all

UTF-16 exists because Unicode was originally supposed to be a fixed-width 16-bit encoding and many systems were designed during this era and needed to be retrofitted to support more characters. These aren't some niche systems or systems that are on their way out, they are major current techologies like windows, .net, Java and QT.

UTF-32 exists because some people think it's easier if each code point is stored in a fixed-size unit. IMO this is largely illusory as there is not a 1:1 mapping between unicode code points and what users would call characters (for example most users would say that "Spın̈al Tap" has 10 characters but it requires 11 unicode codepoints to represent) but it nevertheless exists as a perceived advantage.

I don't think either of these encodings is going to go away any time soon. So if your system interacts widely with other stuff you are likely to end up dealing with other unicode encodings sooner or later.

Will UTF-8 ever become "obsolete" and replaced by "UniversalCode-128" or something, which also includes the alphabets of later discovered nearby galaxies' civilizations?

If we ever establish meaningful contact with intelligent alien life then some decisions would need to be made as to how to represent their languages on our computers and vice-versa. That could eventually mean switching away from computing standards as we know them today to a new set of interplanetary standards.

I think realistically though it's incredibly unlikely that will happen. IMO even if alien-life exists and even if we discover it, it would be impractical to establish meaningful communications without faster than light communications and/or travel and that would mean breaking physics as we know it.

Assuming we don't establish contact with aliens and assuming that we keep using computers that resemble those we use today, it seems unlikely that our text representation systems will be radically changed, it's possible that at some point the codepoint space will be expanded, but I think it's more likely that greater use will be made of combining characters, variant selectors etc to allow new languages to be represented with fewer code-point allocations.

Improve this answer
$\endgroup$
0
2
$\begingroup$

There is a theoretical possibility that over a million code points might not be enough. This is made less likely by the fact that characters can be made from more than one code point, so we could easily reserve one of the 17 pages for "intergalactic languages", where the first code point specifies one of 65,000 languages and the second code point a character in the language.

We might extend UTF-8 to 5 byte characters, but that would break lots of current code that correctly expects 4 byte only.

Why more than 65536 code points? It turned out to be not enough, once more and more languages were added, and Chinese / Japanese characters gettng more complete. Using multiple codepoints to extend the character set as I suggested would be a desparate measure, and even a few non-terrestrial civilisations would likely not require it. Doing this without need would be very wrong. There are emojis consisting of many codepoints, but there is a good reason for that.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Is there any reason that even 56,000 code points wouldn't have been enough if characters beyond that were expressed using variable-length entities or grapheme-clusters with clearly identifiable boundaries? Since grapheme clusters ended up being necessary anyway, what real advantage do multi-word code points have over e.g. having character ranges for start/middle/end of a complex character, and saying that a start+end or start+one middle+end would translate as a 10-bit or 16-bit code point, while start+multiple middles+end would be a grapheme cluster or entity? $\endgroup$ – supercat Jun 14 '20 at 19:28
  • 5
    $\begingroup$ Lots of UTF-8 code originally "correctly" handled up to 6 bytes (as the original Plan9 specification allowed), but then was deliberately crippled to conform to a standard built around UTF-16's limits... $\endgroup$ – user1686 Jun 14 '20 at 21:02
  • 1
    $\begingroup$ I probably don't have specific links spelling it out... However, the limit of U+10FFFF comes from the way UTF-16 uses surrogate pairs to represent codepoints above BMP (1024 high * 1024 low = 0x100000). And because Unicode defines all of UTF-8/UTF-16/UTF-32 as "equally legitimate" (chapter 2), it cannot ever define codepoints above the least common denominator, unless it stops supporting UTF-16. So the Unicode version of UTF-8 is accordingly defined up to 4-byte sequences, and longer sequences are defined as "ill-formed". $\endgroup$ – user1686 Jun 15 '20 at 11:46
  • 2
    $\begingroup$ @phk So for example, RFC 2279 allowed for 6 bytes, but later RFC 3629 restricts to 4 bytes, and e.g. OpenBSD libc was changed to impose the same limit afterwards. Also compare original proposal which allowed 6 bytes vs Unicode 13 ("sequence that does not match the patterns listed in Table 3-7 is ill-formed") which only goes up to 4. $\endgroup$ – user1686 Jun 15 '20 at 11:58
  • 3
    $\begingroup$ It doesn't really make sense to talk about "extending UTF-8 to 5-byte characters"; what you're really talking about is extending Unicode beyond code point U+10FFFF. The limitation on UTF-8 is "do not represent code points that can never exist", not "do not use more than 4 bytes" (without going beyond 4 bytes, you can represent code points U+1100000 to 1FFFFF, but since those don't exist, it would be meaningless). $\endgroup$ – IMSoP Jun 15 '20 at 16:22
1
$\begingroup$

UTF-8 is an elegant way to encode a large range of numbers with a variable number of 8-bit bytes. As long as we don't need more characters than it can represent (unlikely unless the people of year 3000 write entirely with emojis that don't exist yet), there really isn't much reason to switch to another encoding. There is far too much momentum in English-centric computing to warrant an encoding that prioritizes other languages...

...That is, unless we encounter intelligent life and start having to integrate our information systems with theirs. All bets are off at this point. There's no guarantee they chose 8-bit clusters as their primary computing data unit. There's also no guarantee they're using binary or electricity as their primary means of computing. But even if they also used 8-bit bytes with 1 mapped to high voltage and 0 mapped to low, the probability that they created the exact same symbols and corresponding bit-encoding is so unlikely that calling it astronomically unlikely wouldn't cut it.

At that point, there will inevitably be a long negotiation process to develop standard codepoints, hopefully with some ability to bring in more intelligent species' languages later. During this time, there will be dozens of competing standards and the growing pains of changing encodings like those that characterized the 90s and early 00s. After a couple decades, humans and aliens will have it figured out and produce a standard that encodes both species' symbols without undue preferential treatment. A decade or two later, most new software will use that encoding.

Maybe then, we'll finally get rid of the wealth of obsolete ASCII control codes and reassign uppercase letters to higher codepoints in order to make room for alien letters.

Even still, there's always a chance that UTF-8 will still work as a multibyte encoding after the addition of an alien race or two. The main difference is that it won't correspond to the same Unicode assignments. We could also potentially keep our own encodings and then have translation layers between them. Anything can happen with aliens.

Improve this answer
$\endgroup$
0
$\begingroup$

1. Unicode is the standard in all fields

Unicode is the unwreckable standard, and multi-byte UTF-8 with its ASCII subset for most purposes, like HTML, the most compact, even for Asian script when mingled with plain Latin script.

Two-byte UTF-16 has the fixed size advantage: taking the nth 256 bytes from a file form 128 UTF-8 chars. Whereas UTF-8 could have a have a half multi-byte sequence at the block limits. However UTF-16 is a historical error, as Unicode grew out the 16 bit range, and now for many Unicode code points, symbols, need two UTF-16 chars. So its fixed-size advantage is moot.

UTF-32, four bytes per code point, is natural, though Unicode is still in the 3 byte range, and will be for some time. So it is guaranteed to at least waste ¼, and even ¾ for plain Latin script.

UTF-8, UTF-16, UTF-32 do not really compete. In the programming language Java char is UTF-16, String literals are stored in the .class as UTF-8. The latest java versions even allow String, text in Unicode, to store solely internally the text in say ISO-8859-1.

UTF-8 will be the main Unicode Transformation Format for text files.

2. Unicode has flaws

Unicode might be the Esperanto of encodings (like with clever features), but that does not come without flaws. The main one is that there are different code point sequences for the principly the same text; there is no canonical form of Unicode. So é can be one single code point, or two: e and a zero-width ´. Again Java offers conversion in java.text.Normalizer.

Another (minor) flaw under Windows, one can determine when a file content is not in UTF-8, but without reading the content that is not feasible, if the encoding is apriori unknown. But that would hold for any other universal encoding too.

Flaws will not imply a future demise of Unicode. However not without glitches. There might come a time when a canonical form of Unicode becomes obligatory; needing a conversion of existing UTF-8 to say UTF-8C.

3. Chaotic Changes Possible

  • An "UTF-24" could be more politically correct, as with UTF-8 Asian scripts have a serious disadvantage.
  • A redesign of Unicode itself seems academically interesting, and could find its proponents, people favoring something new.

This is counter-balanced by the numerous UTF-8 data: XML in general, jason, general Linux encoding in UTF-8, Windows multiple single-byte encodings (which make UTF-8/UTF-16 the lingua-franca for portable text in many applications).

Conclusion

There is no reason to fear a demise of UTF-8.

I was one of the earlier adapters of UTF-8 in programming. And now keep my projects in UTF-8.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ A few nitpicks: There is no such thing as "Two-byte UTF-16"; the fixed-width encoding is called UCS-2, and UTF-16 has always been variable width. Normalisation isn't something just provided by Java, it's part of the Unicode standard. Similarly, I'm not sure what you mean by "under Windows" when talking about determining file encoding. I also have no idea what this sentence is trying to say: "The latest java versions even allow String, text in Unicode, to store solely internally the text in say ISO-8859-1." $\endgroup$ – IMSoP Jun 16 '20 at 8:29
  • $\begingroup$ @IMSoP First for the other readers to follow your well foiunded info. Indeed UCS-2 wanted to do Unicode in two bytes. and this I misnamed UTF-16 in its early history. Ironically much java code treats UTF-16 char as Unicode, say UCS-2, instead of using code points (Unicode). Normalisation is specified by Unicode. A single canonical preference I believe not. Your comment is full of worthy points. Thanks. $\endgroup$ – Joop Eggen Jun 16 '20 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.