Unicode Transformation Format (UTF)
I’ve been reading about tokenizers in the context of LLMs. While implementing the Byte-pair tokenizer, I went a little deeper into understanding about what UTF is, and why UTF-8 is so popular in today’s world.
UTF is an information exchange format, that can be used for any language / script in the world. This essentially means to say, any character from any language in the world, has a corresponding index or code point that it can be mapped to. The characters can either be encoded in UTF-32, UTF-16 or UTF-8.
Example : To determine the UTF-32, UTF-16 and UTF-8 encodings of the Kannada letter ಕ, check this page : https://www.compart.com/en/unicode/U+0C95
The Unicode code point is U+0C95
These are in the hexadecimal representation, converting to binary —
UTF-32 can represent 2³² characters, this is sufficient to represent all the possible characters in the world. However, this leads to an inefficiencies, since, each character will always use 32 bits. For example, english characters — encoding lower and upper case letters require no more than 7 bits, having to use 32 bits always increases the cost and resources to transmit the message over the wire.
UTF-16, can represent 2¹⁶ characters, each character will use 16 bits, however, this is not sufficient to represent all the characters that exist today. To overcome this, a concept of surrogate characters is used — https://learn.microsoft.com/en-us/windows/win32/intl/surrogates-and-supplementary-characters
UTF-8 on the other hand uses a variable encoding format, which means to say that a given character can be encoded using 1–4 bytes, which is also why it is the preferred encoding today.
However, considering this variable nature, given a bit stream, how would you say for sure what character is represented ?
UTF-8 follows a certain protocol for encoding characters, explained as follows :
Characters having a decimal value < 2⁷, a single byte is used, with a leading 0. For example, letter a, will be represented as 0x61 or a decimal value of 97 (https://www.compart.com/en/unicode/U+0061). If the leading bit is 0, it means to say, this is the only byte the character is represented by.
Characters having decimal values between 2⁷ (128) and 2¹¹ (2048), will require 2 bytes. For example, the latin character È, will be represented as 0xC3 0x88. To indicate that there are 2 bytes in the representation, the first byte has a leading 110 and the second byte has a leading 10 or a continuation sequence (https://www.compart.com/en/unicode/U+00C8)
Further for characters having decimal values between 2¹¹ and 2¹⁶, 3 bytes are used. For example, the Ethiopic Syllable Ha ሀ, is represented as 0xE1 0x88 0x80 (https://www.compart.com/en/unicode/U+1200)
Further for characters having decimal values between 2¹⁶ and 2²¹, 4 bytes are used. For example, the character 𐀂, is represented as 0xF0 0x90 0x80 0x82 (https://www.compart.com/en/unicode/U+10002)
Thus, a maximum of 2²¹ or 20,97,152 can be represented. Today about 11,14,112 values are used, or represent characters.
References :