Target Audience for Review: Programming Language Implementers and Lisp Systems Engineers.
Abstract:
This presentation details a practical demonstration of the Systems Implementation for Common Lisp (SICL) during its bootstrapping phase, utilizing Steel Bank Common Lisp (SBCL) as the host environment. The core focus is on the multi-stage, environment-centric initialization process, which involves defining six distinct first-class global environments (E0 through E5) to handle macro definitions, host objects, bridge objects, and finalized SICL/Ezat's objects, respectively. The demonstration highlights the triplication of the Metaobject Protocol (MOP) and Common Lisp Object System (CLOS) hierarchy across these environments. Critical workflow components, including the sequential loading of external libraries (e.g., Alexandria, Eclector, Cleavir) via ASDF and the sophisticated debugging capabilities utilizing the CLUSO inspector and a SICL backtrace facility, are showcased, confirming SICL’s ability to provide source-level debugging information despite execution being delegated to the host Lisp in certain scenarios.
Creating a Common Lisp Implementation (SICL): A Bootstrapping Demo
0:00 Introduction and Context: Robert Strandh presents a live demonstration of the SICL Common Lisp implementation being bootstrapped within the host environment, SBCL (Steel Bank Common Lisp).
1:13 Bootstrapping Initiation: The process begins by invoking ASDF:load-system on the host, followed by execution of the bootstrapping script. The initial phase involves defining macros by manually setting macro-function for DEFMACRO using host functions, a process temporarily required to enable the execution of subsequent SICL code.
2:49 Environment Structure: The bootstrapping relies on six first-class global environments (E0-E5), each serving a specific role:
E0: Dedicated to macro definitions.
E2: Contains host objects.
E3: Contains "Bridge objects," which are host objects representing SICL objects.
E4/E5: Contain "Ezat’s objects," which represent SICL objects with a direct memory structure (header, rack, class pointer), approximating the final native code representation.
3:48 MOP/CLOS Hierarchy Generation: The entire MOP/CLOS class hierarchy is executed three times: first to create host objects, second to create bridge objects (using the host code), and third, utilizing the bridge objects, to generate the final Ezat’s objects in E5, resulting in a crucial circular class structure.
5:49 Loading External Modules: Following core bootstrap, the system executes a "satiation" phase, traversing all classes to compile effective methods. External modules—including Alexandria, Eclector, Colostrum, and Cleavir—are then loaded. The process currently uses load/eval semantics, which the presenter notes will transition to file compilation semantics for efficiency.
8:46 Object Inspection: The CLUSO inspector (sourced from McLEAN/John Mowing) is used to examine the internal structure of the first-class environments and the SICL objects, confirming the presence of the header and rack structures that define the custom object representation.
11:02 REPL Functionality and Error Handling: A REPL is demonstrated in the E5 environment. A forced error (no applicable method found) is initially caught by the host Lisp (SBCL).
11:53 Advanced Debugging: Even when the error is trapped by the host, the SICL backtrace function (BT), implemented in Clem, successfully generates a backtrace inspector view of the SICL call stack. This tool provides direct source information, referencing the Concrete Syntax Tree (CST) to highlight the specific offending form in the source code.
15:01 Summary of Development Cycle: The development workflow involves iteratively bootstrapping SICL, encountering errors, using specialized debugging tools (CLUSO, BT), fixing code, and restarting the bootstrap.
15:37 ASDF Implementation: The system uses ASDF in two capacities: first, executed by the host (SBCL) to load the initial bootstrapping source code, and second, within SICL’s first-class environments (using a Cleavir-based compiler) to load systems like Alexandria.
17:13 Host Compatibility Considerations: SICL is designed to be compatible with any "complete" Common Lisp implementation (e.g., CCL), but the presenter acknowledges practical limitations encountered with high resource usage, such as SBCL failing due to exhausting immobile memory space.
18:15 Project Insights: The project was characterized as an immense learning experience, highlighting the complexity of implementation details initially underestimated by the developer.