1. Unicode is the standard in all fields
Unicode is the unwreckable standard, and multi-byte UTF-8 with its ASCII subset for most purposes, like HTML, the most compact, even for Asian script when mingled with plain Latin script.
Two-byte UTF-16 has the fixed size advantage: taking the nth 256 bytes from a file form 128 UTF-8 chars. Whereas UTF-8 could have a have a half multi-byte sequence at the block limits. However UTF-16 is a historical error, as Unicode grew out the 16 bit range, and now for many Unicode code points, symbols, need two UTF-16 chars. So its fixed-size advantage is moot.
UTF-32, four bytes per code point, is natural, though Unicode is still in the 3 byte range, and will be for some time. So it is guaranteed to at least waste ¼, and even ¾ for plain Latin script.
UTF-8, UTF-16, UTF-32 do not really compete. In the programming language Java char is UTF-16, String literals are stored in the .class as UTF-8. The latest java versions even allow String, text in Unicode, to store solely internally the text in say ISO-8859-1.
UTF-8 will be the main Unicode Transformation Format for text files.
2. Unicode has flaws
Unicode might be the Esperanto of encodings (like with clever features), but that does not come without flaws.
The main one is that there are different code point sequences for the principly the same text; there is no canonical form of Unicode. So é
can be one single code point, or two: e
and a zero-width ´
. Again Java offers conversion in java.text.Normalizer
.
Another (minor) flaw under Windows, one can determine when a file content is not in UTF-8,
but without reading the content that is not feasible, if the encoding is apriori unknown.
But that would hold for any other universal encoding too.
Flaws will not imply a future demise of Unicode. However not without glitches. There might come a time when a canonical form of Unicode becomes obligatory; needing a conversion of existing UTF-8 to say UTF-8C.
3. Chaotic Changes Possible
- An "UTF-24" could be more politically correct, as with UTF-8 Asian scripts have a serious disadvantage.
- A redesign of Unicode itself seems academically interesting, and could find its proponents, people favoring something new.
This is counter-balanced by the numerous UTF-8 data: XML in general, jason, general Linux encoding in UTF-8, Windows multiple single-byte encodings (which make UTF-8/UTF-16 the lingua-franca for portable text in many applications).
Conclusion
There is no reason to fear a demise of UTF-8.
I was one of the earlier adapters of UTF-8 in programming. And now keep my projects in UTF-8.