Understanding the Unicode Standard (Part 2): A Deep Dive into UTF-8 and UCS-1

UTF-8 is the most widely used encoding system, especially in programming. A sample HTML code will necessarily include a meta tag specifying the encoding used on the page.

<!DOCTYPE html>
<html>
  <head>
    <title>Page Title</title>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
  </head>
  <body>
    <!-- Page content goes here -->
  </body>
</html>

In this article, we will go over the encoding method for UTF-8 and explore some more specifications of Unicode.

How Does UTF-8 Encoding Work?

Here, there are some details to consider:

• UTF-8 encodes characters in eight-bit sequences.
• If a character’s code point requires more than one byte, UTF-8 optimizes the number of bits used.
• UTF-8 encoding can vary from one to four bytes for nearly 1.2 million code points.

UTF-8 ensures optimum memory usage and is backward-compatible with ASCII, meaning the first 128 code points have the same encoded value in both systems.

Encoding with UTF-8

Case 1: Code Point Fits in One Byte

This case corresponds to the 128 ASCII characters. An eight-bit sequence starts with the bit 0.

Example: Encode the NUMBER SIGN (#)

• # => U+0023
• 0023 => (10111)₂

The corresponding UTF-8 code is:

• 00010111

Case 2: Code Point Fits in Two Bytes

If the code point binary format has over seven bits, it will be rewritten as two bytes. The first byte starts with 110xxxxx and the second byte starts with 10xxxxxx.

Example: Encode the ARABIC LETTER KASHMIRI YEH (ؠ)

• ؠ => U+0620
• 0620 => (110 0010 0000)₂ (11 bits)

The corresponding UTF-8 code is:

• 11011000 10100000

Case 3: Code Point Fits in Three Bytes

For code points higher than or equal to 2048, the binary format spans beyond 11 bits and requires three bytes. The first byte starts with 1110xxxx, the second with 10xxxxxx, and the third with 10xxxxxx.

Example: Encode the ETHIOPIC SYLLABLE REE (ሬ)

• ሬ => U+122C
• 122C => (0001 0010 0010 1100)₂ (16 bits)

The corresponding UTF-8 code is:

• 11100001 10001000 10101100

Case 4: Code Point Fits in Four Bytes

For code points beyond 65,535, four bytes are used. The first byte starts with 11110xxx, and the second, third, and fourth bytes start with 10xxxxxx.

Example: Encode the MATHEMATICAL SCRIPT SMALL L (𝓁)

• 𝓁 => U+1D4C1
• 1D4C1 => (0 0001 1101 0100 1100 0001)₂ (21 bits)

The corresponding UTF-8 code is:

• 11110000 10011101 10010011 10000001

Getting Gibberish

If you’ve opened a file with an editor and seen something like this:

#†š®Ùbä_Ò6}9Bo;Y>M šI#ìã£ÊËŒõL§Ê»¾oU²4a(ç¥"1NÖM¬¡ÅU¦4¡...

This happens when you open a file in the wrong encoding. For example, opening a PDF with a code editor like Notepad++.

For normal text written in another language, incorrect encoding can cause gibberish. For example, Japanese text:

こんにちは、私はこれが大好きです (hello, I love this)

When read using default ANSI encoding in Notepad++, it becomes:

こんにちは、私はこれが大好きです

Note on Code Points: The current highest code point for all encoding systems is U+10FFFF. This is not to be confused with the replacement character "�" (U+FFFD) used to represent errors in UTF-8 encodings.

UTF-8 Bias Toward English

UTF-32 does not favor any characters in terms of space consumption, while UTF-16 and UTF-8 do. English remains the least memory-consuming language due to its ASCII roots. Given English's global prevalence in programming and technology, it makes sense to optimize storage for ASCII/ANSI-encoded programs.