Comprehensive Guide to Unicode Standard: Understanding UTF-32 and UTF-16 (Part 1)

Unicode is an open-source standard or a universal character encoding system. It visually defines how most languages in Europe, America, Asia, and some parts of Africa are represented on computer screens. Unicode uses a table of characters with their corresponding code points or binary encoding format. The standard can accommodate nearly 1.2 million individual characters, making it sustainable. Today, over 120,000 symbols are in use, and the complete list of updated characters in the standard can be found here.

Why Unicode Matters: Moving Beyond ASCII Limitations

ASCII is an American standard used to represent the English alphabet characters in a seven-bit format, meaning up to 128 characters can be defined. However, ASCII cannot accommodate characters from other languages, like Japanese kanji or hiragana. Therefore, the Unicode Consortium provided several standards to solve this issue:

UTF-32
UTF-16
UTF-8

Each standard has its own use and encoding/decoding specificities. Today, UTF-8 is the most used encoding, being the most memory-efficient. This article discusses the encoding and decoding processes of UTF-32 and UTF-16, and their efficiency, and includes a mini-lab to experiment with these methods.

For great video resources about Unicode, you can check out these three videos I loved:

UTF-32

UTF-32 is a Unicode consortium standard that works as follows:

Each character is associated with a unique code point.
Each code point is encoded with 32 bits or 4 bytes.

UTF-32 offers an easy way to read characters from memory since every 32 bits represents a unique character. For example, the letter 'a' would be:

Hexadecimal: U+00000061
Binary: 00000000 00000000 00000000 01100001 (04 bytes)

However, UTF-32 is memory-intensive. For example, writing "Hello World" would require 265 bits, making UTF-32 inefficient for memory-constrained systems. For a document with 2000 words averaging five characters per word, UTF-32 would use significantly more storage compared to ASCII.

UTF-16

UTF-16 works as follows:

The first 65,536 characters use 16 bits for representation.
Surrogate pairs are used to represent characters beyond this limit.

So what are surrogate pairs? Let’s consider the following character to encode using UTF-16: 😃 or the Smiling face with open mouth emoji. Its code point is U+1F603 or 128 515 in decimals.

To encode our emoji using UTF-16, we will:

First, we subtract the 65k-ish from the codepoint (U+1F603)

128515 - 65 536 = 62 979

The remainder is converted to binary, then we complete the code to a 20 bits value.

(62 979)10 = (1111011000000011)2 => (0000111101 1000000011)2

then split into two binary codes of 10 bits each

A=> (0000111101)2 => (61)10 & B=>(1000000011)2 => (515)10

Next A & B are associated with surrogate pairs. But what are those?

Note: With surrogate pairs, a Unicode code point from range U+D800 to U+DBFF (called "high surrogate") gets combined with another Unicode code point from range U+DC00 to U+DFFF (called "low surrogate") to generate a whole new character, allowing the encoding of over one million additional characters.

~ Microsoft

Surrogate pairs are two intervals of code point values, reserved for the UTF-16 used case (only applicable here) and used to encode characters with codepoints beyond U+010000. Each interval counts around one thousand values organized as follows:

High surrogates: D800 to DBFF (55296 to 56319)
Low surrogates: DC00 to DFFF (56320 to 57343)

Those are surrogate pairs.

Next, we add D800 (high surrogate) to A (first part of the code) and DC00(low surrogate) to B (second part fo the above code). We then obtain:

A => 61 + 55 296 = 55 357 => ( 1101100000111101 )2

B=> 515 + 56 320 = 56 745 => ( 1101110110101001 )2

These new values are our two 16-bit codepoints representing our emoji.

Note: The surrogates pair allow us to encode all 1 112 064 characters currently that are or will be part of Unicode.

Key Takeaways

UTF-32: Simplifies reading from memory but consumes a lot of memory, making it impractical for most systems and applications.
UTF-16: Encodes the first 65k-ish characters with 16 bits. Beyond this, it uses surrogate pairs, offering better storage usage than UTF-32 but still inefficient for ASCII-only content.

In the next articles, we will dive deeper into the encoding and decoding processes of UTF-8 and its practical applications.