The persona adopted for this summary is that of a Senior Compiler Architect and Language Implementation Specialist.
Abstract:
This presentation details the development of SICL (System for Implementing Common Lisp), a ground-up implementation of the ANSI Common Lisp specification driven by a desire for more idiomatic, CLOS-centric code and reduced cross-implementation divergence. The speaker, Robert Strandh, outlines the core motivations: dissatisfaction with the heavy reliance on implementation-specific constructs and non-idiomatic code in existing systems (like SBCL or ECL).
The evolution of the project moved from a modular layering approach to utilizing the full Common Lisp language for each component, necessitating a complex bootstrapping methodology. This technique requires executing target-specific code within the host environment via first-class global environments to correctly initialize the Common Lisp Object System (CLOS) metaobject protocol (MOP) structures—classes and generic functions—before native compilation can occur. Key technical innovations include optimized generic dispatch (moving beyond old table-driven methods), logarithmic traversal for FROM-END sequence functions, satiation for resolving MOP circularities, path replication, and planned call-site optimization.
Furthermore, the presentation provides an overview of the project's successful extraction of several core components into reusable, implementation-independent libraries, including Eclector (configurable reader), Cleaver (multi-IR compiler framework), Colostrum (first-class environments), and Cluster (object-based assembler). Future work focuses on refining the bootstrapping semantics to support compile-file rather than load, integrating John Mooring's S-Expression Syntax library for robust internal syntax checking, and investigating the relationship between Static Single Assignment (SSA) and Global Value Numbering for advanced register allocation strategies.
Exploring SICL: A Deep Dive into a New Common Lisp Implementation
0:13 Motivation for SICL: Driven by dissatisfaction with existing Common Lisp implementations due to insufficient use of CLOS, excessive implementation-specific code, and code duplication across vendors.
1:22 Project Goal: To implement the full Common Lisp specification from scratch, emphasizing an idiomatic, CLOS-centric design philosophy.
4:59 Rationale for Scratch Development: Modifying existing implementations to meet the required stylistic and structural changes would be prohibitively difficult and likely unacceptable to current maintainers.
7:24 Current Architectural Shift: The project abandoned an initial modular approach (relying on subsets) in favor of using the full language specification, including DEFGENERIC and DEFCLASS, for all modules.
8:38 Bootstrapping Necessity: This design mandates executing target-specific code within the host Lisp environment, necessitating the invention of first-class global environments (see 34:20) to isolate the target state from the host state during compilation setup.
9:35 Key Technical Innovations: A list of major developments since 2008, including:
Optimized generic dispatch using register operations rather than older table-driven techniques (10:53).
Logarithmic traversal for sequence functions when using the FROM-END argument (12:40).
Satiation to resolve circular dependencies during CLOS bootstrapping by preemptively computing effective methods (13:19, 23:55).
Path replication to avoid redundant type checks in control flow (13:32).
Planned Call-Site Optimization to eliminate keyword argument parsing overhead (14:44).
17:29 Critique of PCL: The reliance on Portable Common Loops (PCL) in many systems introduces unidiomatic special cases that the SICL design explicitly seeks to avoid.
19:21 CLOS Defined by Execution: The MOP structure (slots of STANDARD-CLASS, effective methods) is best derived by executing the DEFCLASS/DEFMETHOD forms, which necessitates host-specific code execution during bootstrapping (20:55).
22:09 AST Traversal: The bootstrapping process involves compiling SICL code to an AST, then to a control graph IR, where each instruction is executed via a host closure to affect the evolving target environment.
24:34 Extracted Modules (General Principle): Components deemed sufficiently general are being extracted to stand-alone, implementation-independent libraries (25:04).
26:48 Concrete Syntax Tree (CST): Classes used to wrap S-expressions, specifically to attach metadata like source location, solving symbol identity issues across source occurrences (27:13).
27:58 Eclector (Reader): A highly configurable reader (derived from the original SICL reader) designed to preserve normally discarded information like comments and reader macro results, critical for text editors (28:29).
30:05 Cleaver (Compiler Framework): A multi-stage compiler generating code via sequential IRs: AST (macro expansion) $\rightarrow$ High-Level IR (CL objects only) $\rightarrow$ Medium-Level IR (introduces memory ops) $\rightarrow$ Low-Level IR (introduces registers) $\rightarrow$ Code Generation (31:53).
32:32 Cleaver Versions: Two versions exist: the stable version in the SICL repo and the actively developed, extracted version used by Clasp (32:46).
34:20 Colostrum (First-Class Environments): The implementation layer for first-class environments, designed to solve library dependency conflicts (e.g., needing LibA v1 and v2 simultaneously) via environment isolation (35:12).
38:01 Cluster (Assembler): Designed abstractly to accept standardized instruction objects rather than surface text syntax, allowing the compiler to generate binary code directly (39:06).
40:00 Bootstrapping Complexity: The bootstrapping procedure is extremely complex, requiring an evolution from simple LOAD semantics to full COMPILE-FILE semantics to handle environment transfers correctly (41:03).
44:39 Future Work: S-Expression Syntax Library: Plan to replace all internal, custom syntax-checking code (for special forms, declarations, etc.) with John Mooring's external, robust library (44:44).
46:52 Future Work: SSA/GVN Investigation: Investigating whether Global Value Numbering (GVN) is a superset of Static Single Assignment (SSA) to potentially allow the register allocator to make better cost-based decisions regarding re-materialization versus memory access (48:48).
51:00 Future Work: Claw Sauce: A long-term project concept for an operating system built around the isolation provided by first-class global environments (51:00).
51:23 Future Work: Second Climax: Integrating Cleaver passes into the Climax editor for real-time compilation feedback as the user types (51:23).
51:56 Call for Maintainers: Active solicitation for contributors for modules like Colostrum, Truckler, and for extracting Loop and Format (52:03).