Jacob Moody's presentation, "Modern Unicode Requires Modern Solutions," details the ongoing efforts within the Ninefront operating system to fully support modern Unicode standards, particularly 21-bit characters and normalization. Starting with historical context on Plan 9's early adoption of UTF-8, the talk highlights the challenge of updating core library functions (like character casing) that were not designed for the expanded Unicode planes. Moody describes the development of a code-generation system that leverages Unicode's published data files to create efficient, constant-time lookup tables using "compact arrays" and hash tables, significantly improving performance and maintainability compared to older binary search methods. A major focus is the implementation of Unicode normalization (NFC and NFD forms), addressing the issue of multiple byte representations for visually identical characters. This involved complex data structures for decomposition and recomposition, along with a novel streaming algorithm to handle normalization within bounded memory, validated by extensive testing against Unicode's own test suite. The presentation concludes by discussing the immediate benefits (e.g., in text conversion and string libraries) and outlines significant future work, particularly in updating graphical user interfaces and font systems to properly handle decomposed Unicode characters and complex text shaping.
Modern Unicode Requires Modern Solutions: An Overview of Ninefront's Unicode Implementation
0:01 Introduction & Speaker Background: Jacob Moody, a growing Unicode enthusiast, discusses his work on Ninefront, focusing on Unicode support and porting.
0:30 Historical Context of UTF-8 in Plan 9: Plan 9 was an early adopter of Unicode (V1, 1991). UTF-8 was designed by Ken Thompson and implemented by Rob Pike in less than a week, eventually contributing to its standardization.
1:51 Unicode Evolution & Ninefront's Challenges: Unicode expanded to 21-bit characters (V2 onwards), a significant change from the initial 16-bit goal. Plan 9 was the first to implement 21-bit codepoint support. Ninefront continues to update its software for full 21-bit compatibility, an ongoing effort.
3:18 Outdated Character Functions: Ninefront's liblibc character type functions (e.g., two_upper_rune, is_lower_rune) were not updated beyond the 16-bit Basic Multilingual Plane (BMP), leading to missing support for many modern characters.
6:36 Efficient Lookup Table Design: To address performance and maintainability issues of a growing static array with binary search, a code-generation approach was adopted. This utilizes "compact arrays" (a form of page table lookup) to efficiently store sparse Unicode data for constant-time lookups.
11:26 Implementation Results for Casing Functions: The casing functions (two_upper_rune, is_lower_rune, etc.) were implemented using compact arrays, resulting in a small memory footprint (approx. 60KB total) for constant-time character transformations.
13:30 The Normalization Problem: Unicode allows for multiple byte representations of the same visual character (e.g., precomposed vs. decomposed forms). This requires "normalization" (NFC for composed, NFD for decomposed) to ensure semantic consistency, crucial for operations like file paths and logins.
17:14 Data Structures for Normalization: Implementing normalization required new data tables: decomposition/recomposition mappings, combining character class (for sorting), and quick check values (for boundary detection in streaming). Hash tables were used for efficient recomposition lookups.
26:55 Streaming Normalization Algorithm: A key challenge was to implement normalization efficiently for streaming input without requiring infinite memory. This was achieved by chunking the input based on character contexts and using a Combining Grapheme Joiner (U+034F) to handle excessively long decomposed sequences.
33:17 Library Interface & Testing: A streaming API (norm_pull) was developed, along with fixed-size convenience functions. The implementation was rigorously tested against tens of thousands of Unicode standard test cases and custom edge-case tests.
37:24 Current Adoption: The new Unicode normalization support has been integrated into Ninefront's Text Conversion System (TCS) and is being used by the streetlist project.
38:24 Additional "Free" Algorithms: The developed data tables also enabled trivial implementation of grapheme_break and word_break algorithms, which are useful for text editing and display.
39:10 Future Work & Challenges: The next major hurdle is updating Plan 9's graphical user interfaces (Acme, SAM, RIO) and its font system to properly handle decomposed Unicode, as they currently assume a one-to-one mapping between codepoints and characters. Defining consistent user experience for actions like backspacing on decomposed characters is also an open question.
43:49 Q&A - One-Way Decomposition: The speaker explains "singleton decompositions" where a single codepoint decomposes to another but cannot recompose back, often due to legacy character handling.