The ideal group of people to review this topic is Senior Common Lisp Developers and Package Management Specialists.

Abstract:

This presentation introduces Qlot (also referred to as kot), a project-local library installer designed to address critical dependency management deficiencies within the Common Lisp ecosystem, primarily stemming from limitations in Quicklisp. The core issues Qlot solves are the inability to easily install specific or non-distribution versions of libraries, and the lack of robust environment isolation and reproducibility across different deployment targets (development, production, CI).

Qlot operates by creating a dedicated, project-local Quicklisp distribution for each project, ensuring isolation from other Lisp projects on the same machine. Configuration is managed through Qfile and the lock file Qfile.log, analogous to tools like Ruby's Bundler. Crucially, Qlot employs specialized parsing techniques—leveraging Quicklisp metadata rather than relying on standard ASDF functionality—to determine project dependencies. This approach breaks a circular dependency problem that arises during the initial setup of a local Quicklisp environment, particularly when ASDF files themselves execute arbitrary Lisp code. While the command-line interface is robust, the integration into the Common Lisp REPL environment remains experimental due to the architectural necessity of spawning external processes for complex operations (e.g., HTTPS support).

Qlot, a Project-Local Library Installer

• 0:05 Common Lisp Dependency Problem: The author, a Common Lisp web developer, frequently encounters issues stemming from Quicklisp's lack of proper dependency management, leading to problems like systems being unfound or running incompatible library versions.
• 3:58 Quicklisp Limitations: Quicklisp lacks two primary features: (1) A function to install newer or different library versions outside of the distribution's current index, and (2) easy reproducibility of the exact set of libraries across environments.
• 4:47 Reproducibility Requirement: Reproducibility is crucial for projects involving multiple developers, various deployment environments (production, staging), or CI/CD pipelines.
• 6:06 Submodule Critique: Using Git submodules for dependency management is deemed a partial and insufficient solution. While it avoids introducing new tools, it is hard to maintain (requires manually listing all transitive dependencies) and fails to provide environment isolation.
• 8:00 Isolation Necessity: Environment isolation is necessary when different projects on the same machine require conflicting versions of the same library (e.g., Project A needs Alexandria v1, Project B needs Alexandria v2).
• 10:02 Introducing Qlot (kot): Qlot is presented as a tool to set up Quicklisp locally for each project, track installed library versions, enable reproducibility, and isolate environments. It also supports HTTPS transport.
• 11:23 Qlot Initialization and Installation: Basic setup involves kot init (creates dependency management files) and kot install (downloads and sets up the local Quicklisp distribution in the .qlot directory).
• 11:53 Execution: The project-local Quicklisp is accessed using kot exec [command], which runs a new Lisp process (e.g., SBCL) with the isolated dependencies.
• 12:26 Lisp Implementation Support: Qlot supports popular implementations like SBCL, ECL, ABCL, CCL, Clasp, and Allegro CL.
• 13:52 Adding Dependencies: The kot add command is used to introduce new dependencies, supporting installations from the latest Quicklisp dist or specific versions, as well as upstream Git repositories via URL, branch, tag, or commit hash.
• 15:35 Configuration Files: Dependencies are managed via Qfile (user-desired state) and Qfile.log (detailed, system-reproducible information, including repository URLs and commit references). Both files must be committed to version control.
• 16:59 Updating: kot update refreshes the project dependencies to the latest version and overwrites Qfile.log. Other users must run kot install to apply the changes.
• 18:08 Internal Design Principle: The design principle is "good design is invisible," implying complex internal mechanisms are abstracted away from the user.
• 18:40 Quicklisp Metadata Requirement: Quicklisp dists require specific metadata files, notably releases.txt (list of systems) and systems.txt (dependency graph).
• 20:08 Circular Dependency Problem: Qlot cannot use standard ASDF functionality (like ASDF:find-system) to determine dependencies during setup because ASDF loads the system definition files (.asd), which can contain arbitrary Lisp code that might load external libraries (e.g., CFFI). This creates a circular dependency: Qlot needs the dependencies to set up the local environment, but the environment must be set up before it can safely load the definition files.
• 21:59 Solution (Non-ASDF Parsing): Qlot avoids this by using qlot:working-forms and calculating dependencies based on pre-generated Quicklisp metadata (systems.txt), which is significantly faster than parsing ASDF systems directly.
• 23:23 REPL Interface (Experimental): The primary interface is CLI. The REPL interface is experimental because Qlot needs external libraries (like those for HTTPS) that cannot run in the same process as the user's application without risking conflicts, necessitating separate process execution for complicated tasks.
• 28:06 Inspiration and Alternatives: Qlot’s subcommands are heavily inspired by Ruby’s Bundler. It is functionally similar to Eric Timmons' CLPM (Common Lisp Package Manager), but Qlot integrates with and leverages Quicklisp, whereas CLPM aims to replace it entirely.
• 26:06 Q&A: Environment Sharing: Qlot explicitly creates a distinct Quicklisp distribution for every project, ensuring maximal isolation, even if dependency sets are compatible.
• 31:54 Q&A: Patching and Forks: Qlot supports specifying custom forks or patched versions of dependencies via Git branch, tag, or commit reference options.