UTF-8
From Wikipedia, the free encyclopedia
UTF-8 (8-bit UCS/Unicode Transformation Format) is a variable-length character encoding for Unicode. It is able to represent any character in the Unicode standard, yet the initial encoding of byte codes and character assignments for UTF-8 is backwards compatible with ASCII. For these reasons, it is steadily becoming the preferred encoding for e-mail, web pages[1], and other places where characters are stored or streamed.
UTF-8 encodes each character (code point) in one to four octets (8-bit bytes), with the 1-byte encoding used for the 128 US-ASCII characters. See the Description section below for details.
Four bytes may seem like a lot for one character . However UTF-16 (the main alternative to UTF-8) also uses four bytes for these same code points. Whether UTF-8 or UTF-16 is more efficient depends on the range of code points being used. For English using ASCII UTF-8 will be 1/2 the size, for all other Western languages UTF-8 will range from 1/2 size to trivially smaller, depending on the number of ASCII characters such as spaces, numbers, and unaccented ASCII letters, are in the text. For Chinese it UTF-8 can be larger if less than 50% of the characters are spaces, numbers, newlines, tabs, and other ASCII characters. It is important to realize that all the standard Unicode encoding schemes are not intended for compression, modern algorithms such as used by gzip will compress any of them to approximately the same size and typically much smaller than 1 byte per character. For short items of text where traditional algorithms do not perform well and size is important, the Standard Compression Scheme for Unicode could be considered instead.
The Internet Engineering Task Force (IETF) requires all Internet protocols to identify the encoding used for character data, and the supported character encodings must include UTF-8.[2] The Internet Mail Consortium (IMC) recommends that all email programs be able to display and create mail using UTF-8.[3]
Contents
[edit] History
By early 1992 the search was on for a good byte-stream encoding of multi-byte character sets. The draft ISO 10646 standard contained a non-required annex called UTF that provided a byte-stream encoding of its 32-bit characters. This encoding was not satisfactory on performance grounds, but did introduce the notion that bytes in the ASCII range of 0–127 represent themselves in UTF, thereby providing backward compatibility.
In July 1992 the X/Open committee XoJIG was looking for a better encoding. Dave Prosser of Unix System Laboratories submitted a proposal for one that had faster implementation characteristics and introduced the improvement that 7-bit ASCII characters would only represent themselves; all multibyte sequences would include only 8-bit characters, i.e. those where the high bit was set.
In August 1992 this proposal was circulated by an IBM X/Open representative to interested parties. Ken Thompson of the Plan 9 operating system group at Bell Laboratories then made a crucial modification to the encoding, to allow it to be self-synchronizing, meaning that it was not necessary to read from the beginning of the string in order to find character boundaries. Thompson's design was outlined on September 2, 1992, on a placemat in a New Jersey diner with Rob Pike. The following days, Pike and Thompson implemented it and updated Plan 9 to use it throughout, and then communicated their success back to X/Open.[4]
UTF-8 was first officially presented at the USENIX conference in San Diego, from January 25–29 1993.
[edit] Description
The bits of a Unicode character are distributed into the lower bit positions inside the UTF-8 bytes, with the lowest bit going into the last bit of the last byte:
U+000000-U+00007F 0xxxxxxxU+000080-U+0007FF 110xxxxx 10xxxxxxU+000800-U+00FFFF 1110xxxx 10xxxxxx 10xxxxxxU+010000-U+10FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxxSo the first 128 characters (US-ASCII) need one byte. The next 1920 characters need two bytes to encode, this includes Latin letters with diacritics and characters from Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic, Syriac and Thaana alphabets. Three bytes are needed for the rest of the Basic Multilingual Plane (which contains virtually all characters in common use). Four bytes are needed for characters in the other planes of Unicode, which are rarely used in practice.
By continuing the pattern given above it is possible to deal with much larger numbers. The original specification allowed for sequences of up to six bytes covering numbers up to 31 bits (the original limit of the universal character set). However, UTF-8 was restricted by RFC 3629 to use only the area covered by the formal Unicode definition, U+0000 to U+10FFFF, in November 2003.
With these restrictions, bytes in a UTF-8 sequence have the following meanings. The ones marked in red can never appear in a legal UTF-8 sequence. The ones in white must only appear as the first byte in a multi-byte sequence, and the ones in orange can only appear as the second or later byte in a multi-byte sequence:
Unicode also disallows the 2048 code points U+D800..U+DFFF (the UTF-16 surrogate pairs) and also the 32 code points U+FDD0..U+FDEF (noncharacters) and all 34 code points of the form U+xxFFFE and U+xxFFFF (more noncharacters). See Table 3-7 in the Unicode 5.0 standard. UTF-8 reliably transforms these values, but they are not valid scalar values in Unicode, and thus the UTF-8 encodings of them may be considered invalid sequences.
There are several current definitions of UTF-8 in various standards documents:
They supersede the definitions given in the following obsolete works:
They are all the same in their general mechanics, with the main differences being on issues such as allowed range of code point values and safe handling of invalid input.
[edit] Naming
The official name is "UTF-8", which is used in all the documents relating to the encoding. There are many instances, particularly for documents to be transmitted across the internet, where the character set in which the document is encoded is declared by the name near the start of the document. In this case, the correct name to use is "UTF-8". In addition, all standards conforming to the Internet Assigned Numbers Authority (IANA) list[5], which include CSS, HTML, XML, and HTTP headers[6] may also use the name "utf-8", as the declaration is case insensitive. Despite this, alternative forms, usually "utf8" or "UTF8", are seen; while this is incorrect and should be avoided, most agents such as browsers can understand this.
[edit] Examples
The Dollar Sign ($), which is Unicode U+0024 or binary 10 0100:
U+0000 through U+007F The first line of the table shows it will be encoded using one byte, 0xxxxxxx Putting the binary right-justified into the 'x' bits results in 00100100 This byte in hexadecimal is 0x24. Thus the ASCII dollar sign is encoded unchanged.The Cent Sign (¢), which is Unicode U+00A2 or binary 1010 0010:
U+0080 through U+07FF The second line of the table shows it will be encoded using two bytes, 110xxxxx,10xxxxxx. Putting the binary right-justified into the 'x' bits results in 11000010,10100010 These bytes in hexadecimal are 0xC2,0xA2. That is the encoding of the character Cent Sign (¢) in UTF-8.The character aleph (×), which is Unicode U+05D0 or binary 101 1101 0000:
U+0080 through U+07FF The second line of the table shows it will be encoded using two bytes, 110xxxxx,10xxxxxx. Putting the binary right-justified into the 'x' bits results in 11010111,10010000 These bytes in hexadecimal are 0xD7,0x90. That is the encoding of the character aleph (×) in UTF-8.The Euro symbol (€), which is Unicode U+20AC or binary 10 0000 1010 1100:
U+0800 through U+FFFF The third line of the table shows it will be encoded using three bytes, 1110xxxx,10xxxxxx,10xxxxxx. Putting the binary right-justified into the 'x' bits results in 11100010,10000010,10101100 These bytes in hexadecimal are 0xE2,0x82,0xAC. That is the encoding of the Euro symbol (€) in UTF-8.[edit] Rationale behind UTF-8's design
As a consequence of the design of UTF-8, the following properties of multi-byte sequences hold:
0. The most significant bits of the first byte of a multi-byte sequence determine the length of the sequence. These most significant bits are 110 for two-byte sequences; 1110 for three-byte sequences, and so on. The remaining bytes in a multi-byte sequence have 10 as their two most significant bits. A UTF-8 stream contains neither the byte FE nor FF. This makes sure that a UTF-8 stream never looks like a UTF-16 stream starting with U+FEFF (Byte-order mark)UTF-8 was designed to satisfy these properties in order to guarantee that no byte sequence of one character is contained within a longer byte sequence of another character. This ensures that byte-wise sub-string matching can be applied to search for words or phrases within a text; some older variable-length 8-bit encodings (such as Shift-JIS) did not have this property and thus made string-matching algorithms rather complicated. Although this property adds redundancy to UTF-8–encoded text, the advantages outweigh this concern; besides, data compression is not one of Unicode's aims and must be considered independently. This also means that if one or more complete bytes are lost due to error or corruption, one can resynchronize at the beginning of the next character and thus limit the damage.
Also due to the design of the byte sequences, if a sequence of bytes supposed to represent text validates as UTF-8 then it is fairly safe to assume it is UTF-8. The chance of a random sequence of bytes being valid UTF-8 and not pure ASCII is 3.1% for a 2 byte sequence, 0.39% for a 3 byte sequence and even lower for longer sequences.
While natural languages encoded in traditional encodings are not random byte sequences, they are also unlikely to produce byte sequences that would pass a UTF-8 validity test and then be misinterpreted. For example, for ISO-8859-1 text to be misrecognized as UTF-8, the only non-ASCII characters in it would have to be in sequences starting with either an accented letter or the multiplication symbol and ending with a symbol. Pure ASCII text would pass a UTF-8 validity test and it would be interpreted correctly because the UTF-8 encoding for the same text is the same as the ASCII encoding.
The bit patterns can be used to identify UTF-8 characters. If the byte's first hex code begins with 0–7, it is an ASCII character. If it begins with C or D, it is an 11-bit character (expressed in two bytes). If it begins with E, it is 16-bit (expressed in 3 bytes), and if it begins with F, it is 21 bits (expressed in 4 bytes). 8 through B cannot be first hex codes, but all following bytes must begin with a hex code between 8 through B. Thus, at a glance, it can be seen that "0xA9" is not a valid UTF-8 character, but that "0x54" and "0xE3 0xB4 0xB1" are valid UTF-8 characters.
There is no good validity test for most traditional 8-bit encodings like ISO-8859-1. It must be known otherwise which encoding is used, otherwise bad text will be shown. This is called mojibake and other names. The fact that there is a working validity test for UTF-8-encoded texts is a big advantage.
[edit] UTF-8 derivations
The following implementations are slight differences from the UTF-8 specification. They are incompatible with the UTF-8 specification.
[edit] CESU-8
Many pieces of software added UTF-8 conversions for UCS-2 data and did not alter their UTF-8 conversion when UCS-2 was replaced with the surrogate-pair supporting UTF-16. The result is that each half of a UTF-16 surrogate pair is encoded as its own 3-byte UTF-8 encoding, resulting in 6 bytes rather than 4 for characters outside the Basic Multilingual Plane. Oracle databases use this, as well as Java and Tcl as described below, and probably a great deal of other Windows software where the programmers were unaware of the complexities of UTF-16. Although most usage is by accident, a supposed benefit is that this preserves UTF-16 binary sorting order when CESU-8 is binary sorted.
Because CESU-8 and derivations are not UTF-8, one needs to be very careful to avoid mislabelling data in it as UTF-8 when interchanging information over the Internet.
[edit] Modified UTF-8
Modified UTF-8 treats surrogate pairs as in CESU-8, described above. In addition the null character (U+0000) is encoded as 0xc0 0x80 rather than 0x00. (0xc0 0x80 is not legal standard UTF-8 because it is not the shortest possible representation.) This means that the encoding of an array of Unicode containing the null character will not have a null byte in it, and thus will not be truncated if processed in a language such as C using traditional ASCIIZ string functions.
In normal usage, the Java programming language supports standard UTF-8 when reading and writing strings through InputStreamReader and OutputStreamWriter. However it uses modified UTF-8 for object serialization, for the Java Native Interface, and for embedding constants in class files.
Tcl also uses the same modified UTF-8 as Java for internal representation of Unicode data.
[edit] UTF-8 nuances
The following are slight differences between typical UTF-8 usage scenarios, and conform to the UTF-8 specification.
[edit] Windows
Many Windows programs (including Windows Notepad) add the byte sequence EF BB BF—the UTF-8 encoding of the Unicode Byte Order Mark—to the beginning of any document saved as UTF-8. This causes interoperability problems with software that does not expect the BOM. In particular:
Some Windows software (including Notepad) will sometimes misidentify plain ASCII documents as UTF-16LE if there is no BOM, a bug commonly known as "Bush hid the facts" after a particular phrase that can trigger it.
[edit] Mac OS X
The Mac OS X Operating System uses canonically decomposed Unicode. The file names are encoded as UTF-8 in the Mac OS filesystem. In canonically decomposed Unicode, the use of precomposed characters is forbidden and combining diacritics must be used to replace them. It is valid for UTF-8 to be in either NFC or NFD, but most other operating systems expect the shorter NFC form of UTF-8, which can cause an interoperability problem for some applications. This difference in normalization forms is not specific to the Mac.
A common argument[citation needed] is that this makes sorting far simpler, but this argument is easily refuted[citation needed]: for one, sorting is language dependent (in German, the ä character sorts just after the a character, while in Swedish ä sorts after z). Therefore, it can be confusing for software built around the assumption that precomposed characters are the norm and combining diacritics are only used to form unusual combinations. This is an example of the NFD (Normalization Form Canonical Decomposition) variant of Unicode normalization—most other platforms, including Windows and Linux, use the NFC (Normalization Form Canonical Composition) form of Unicode normalization, which is also used by W3C standards, so NFD data must typically be converted to NFC for use on other platforms or the Web.
This is discussed in Apple Q&A 1173[7].
[edit] Overlong forms, invalid input, and security considerations
The exact response required of a UTF-8 decoder on invalid input is not uniformly defined by the standards. In general, there are several ways a UTF-8 decoder might behave in the event of an invalid byte sequence:
It is possible for a decoder to behave in different ways for different types of invalid input.
RFC 3629 states that "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."[8]The Unicode Standard requires a Unicode-compliant decoder to "…treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence."
Overlong forms are one of the most troublesome types of UTF-8 data. The current RFC says they must not be decoded, but older specifications for UTF-8 only gave a warning, and many simpler decoders will happily decode them. Overlong forms have been used to bypass security validations in high profile products including Microsoft's IIS web server. Therefore, great care must be taken to avoid security issues if validation is performed before conversion from UTF-8, and it is generally much simpler to handle overlong forms before any input validation is done.
Another problem is that decoders do not agree on the number of bytes in an invalid sequence. The sequence 0xF0,0x20,0x20,0x20 might be considered a single 4-byte error by some decoders, and a 1-byte error followed by 3 space characters in other decoders. Generally the second interpretation is better, as fewer characters will be lost if a byte is lost or corrupted.
To maintain security in the case of invalid input, there are a few options. The first is to decode the UTF-8 before doing any input validation checks. The second is to use a decoder that, in the event of invalid input either returns an error or text that the application knows to be harmless. A third possibility is to not decode the UTF-8 at all, this is quite practical if the system only treats some ASCII characters (like slash and NUL) specially, and treats all other bytes as identifiers or other data but requires care to avoid passing invalid UTF-8 to other code (such as third party libraries or an operating system) that cannot safely handle it.
[edit] Advantages and disadvantages
[edit] A note on string length and character indexes
A common criticism from beginners of variable-length encoding such as UTF-8 is that the algorithm to find the number of characters between two points, or to advance a pointer by n characters, are not O(1) (constant time), causing programs using them to be slower. However the use of these algorithms by actual working software is often vastly over-estimated:
malloc(strlen(s)+1) or pointer+=length_of_word(*pointer). Changing the functions to return byte count in place of character count will get the exact same program with O(1) performance. It is wrong to assume the number of characters will assist in getting the visual space needed to display a string. Combining characters, double width characters, proportional fonts, non-printing characters and right-to-left characters all contribute to the impossibility of accurately calculating the layout space without analyzing the actual characters being used. The self-synchronizing nature of UTF-8 can be used to find the code point boundary that is near a pointer. This is due to the lead and trail bytes using a disjoint numerical range of possible byte values.So while the number of octets in a UTF-8 string or substring is related in a more complex way to the number of code points than for UTF-32, it is very rare to encounter a situation where this makes a difference in practice, and this cannot be used as either an advantage or disadvantage of UTF-8.
