Expert Persona: Senior Lisp Systems Architect and Codebase Morphologist.
Abstract:
This presentation details the creation and initial analysis of the "Quickref Cohort," a comprehensive dataset derived from the entire Quicklisp ecosystem, achieved through a necessary internal architectural redesign of the DEET reference manual generator. The core motivation stemmed from addressing technical limitations—namely, the impossibility of loading all Quicklisp libraries in a single Lisp image for documentation processing—and external pressures for flexible documentation output formats beyond TeXinfo.
The solution involves transitioning DEET from a monolithic application to a three-stage pipeline, the first stage of which, Assessment (DEET 4.0 beta 1), generates a format-independent, introspectively derived data structure called a Report. By dumping these Reports for over 2,000 libraries, the Cohort was created, containing over 500,000 programmatic definitions.
Initial quantitative morphological analysis of the Cohort reveals key statistical insights into the Common Lisp codebase, including dominant symbol length distributions (10-15 characters) and notably low documentation rates (averaging one-third documented, with methods and slots showing significantly lower adherence). The presenter concludes by soliciting expertise from the Lisp community regarding complex serialization strategies necessary to fully dump the Reports, ensuring cross-reference preservation while excluding transient Lisp object pointers.
The Quickref Cohort: Analysis and Redesign of the DEET Documentation Engine
0:00 Project Context: The discussion focuses on side projects originating from the Quickref DEET project, which aims to document the entire Quicklisp library world.
0:32 Defining "Cohort": In the academic context relevant to the project, a cohort is defined as a substantial database of information (e.g., programmatic definitions) intended for experimental studies or gaining statistical insights.
2:33 DEET and Quickref History: DEET (reference manual generator) began in 2010. Quickref, initiated in 2017, wraps DEET to automatically generate reference manuals for over 2,000 Quicklisp libraries, leveraging introspection.
4:04 Internal Design Constraint (Pressure Point 1): Quickref requires running DEET in separate Lisp images for each library because loading the entire Quicklisp world into one environment is impractical due to size constraints and conflicting library dependencies.
5:53 External Demand (Pressure Point 2): There is user demand for direct HTML generation with flexible styling, moving away from DEET’s current reliance on the TeXinfo intermediate format.
7:15 DEET Architectural Redesign: The engine is being revamped into a three-stage pipeline:
Assessment (9:31): Loads the ASDF system and performs introspection to create a raw internal data structure called a Report, complete with exhaustive cross-references. (Completed in DEET 4.0 beta 1).
Assembly (8:34): Organizes the Report data (e.g., separating public and private interfaces). (Pending implementation).
Type Setting (9:15): Generates the final, human-readable documentation (e.g., HTML, PDF). (Pending implementation).
10:48 Cohort Creation: The ability to generate the standardized Report data structure in Stage 1 led to the decision to dump these Reports for all Quicklisp libraries, creating the Quickref Cohort for morphological analysis.
11:25 Cohort Scale: The dataset currently encompasses more than 500,000 programmatic definitions (classes, macros, methods, etc.). The beta version is available for download on the Quicklisp website.
14:43 Preliminary Data Morphology: Current dumps focus on quantitative metrics (e.g., length of docstrings, number of exported symbols, component counts) rather than full object serialization.
16:14 Symbol Length Distribution: Statistical analysis shows that most symbols in Quicklisp have a length between 10 and 15 characters.
17:23 Documentation Rates: Approximately one-third (33%) of all programmatic definitions in Quicklisp are currently documented. Documentation rates are particularly low for methods, slots, and compiler macros.
19:04 Class Morphometrics: Analysis of class-of objects shows classes average roughly 8 methods and slightly more than 2 slots. The low average number of direct superclasses (just over 1) indicates infrequent utilization of multiple inheritance within the Quicklisp ecosystem.
21:13 Outlier Data Records: The cohort contains extreme outliers, including a doc string of nearly 30,000 characters used as a full user manual for a macro, and a programmatically generated function name 135 characters in length.
24:57 Serialization Requirement: The project requires a robust serialization strategy to perform a "serious dump" of the complete Reports, specifically needing guidance on preserving the complex network of cross-references between programmatic definitions while safely discarding transient Lisp object pointers (e.g., when handling eql specializers in lambda lists).
27:11 Documentation Gaps: Introspection-based extraction means documentation embedded exclusively in source code comments is currently unavailable, as comments are discarded upon library loading.
29:33 Future Tooling: The Cohort data is anticipated to facilitate the creation of complex tools such as dependency graphs (for classes, conditions) and potentially optimized test runners.