This project details the architecture and implementation of a novel computational system designed for "syntopic reading," moving beyond conventional LLM summarization to discover deep, non-obvious thematic connections ("trails") between excerpts from a corpus of 100 non-fiction books. The core system utilizes Claude Code as a senior agent, equipped with a bespoke Command Line Interface (CLI) to browse a highly structured knowledge base. The architecture involves chunking text (approx. 500 words), extracting topics using Gemini 2.5 Flash Lite, indexing topics and embeddings (via SQLite and sqlite-vec), and organizing the knowledge into a hierarchical topic tree via recursive Leiden partitioning. A critical finding during development was the superior performance of an agent-driven workflow, where development focus shifted from prompt engineering to enhancing the agent’s toolset and affordances. The system incorporates a novelty bias in its search algorithms to push exploration into under-represented thematic areas. While the technical execution is robust, community feedback highlights a division over the conceptual quality and depth of the LLM-generated connections.
Computational Linguistics and Agentic Workflow Analysis
Project Mandate: The primary objective is to leverage advanced LLMs (specifically Claude Code) to perform syntopic reading—finding thematically connected "trails" of excerpts across a library of 100 non-fiction books—as an alternative to traditional, reductive summarization.
Data Preparation and Indexing: The corpus, sourced from Hacker News favorites, was segmented into approximately 500-word, paragraph-aligned chunks. Gemini 2.5 Flash Lite was used to extract 3–5 topics per chunk, resulting in over 100,000 topics indexed for search. The total indexing process consumed roughly 60 million input tokens, costing about £10.
Topic Hierarchy Generation: The vast collection of topics was made browsable by constructing a hierarchical tree structure. This involved generating a graph based on embedding similarity and point-wise mutual information of topic co-occurrence, followed by recursive Leiden partitioning to create approximately 1,000 top-level clusters.
Agentic Control Flow: Claude Code operates through a CLI offering tools to search topics, find co-occurring topics within a chunk window, and navigate the topic hierarchy. This agent performs a multi-stage process: generating novel trail ideas, researching specific trails by browsing chunks, and extracting/ordering excerpts to support a central insight.
Workflow Optimization (Agentic vs. Pipeline): The author observed a significant efficiency and quality improvement when transitioning from a rigid, multi-stage LLM pipeline to a flexible, tool-driven agent model. The agent demonstrated an ability to self-infer necessary CLI calls faster than manual scripting.
Development Paradigm Shift: The core development focus moved from optimizing prompts to improving the functionality and affordances of the tools available to the agent, enabling complex, automated tasks like the global editing and rebalancing of existing trails.
Guiding Search via Novelty: To prevent convergence on common themes, the search algorithm introduced a novelty metric. This score was calculated as the mean distance of a topic's embedding from its k-nearest neighbors, biasing search results towards under-explored areas of the corpus.
Observed Conceptual Bias: The agent showed a recurrent tendency to fixate on themes of secrecy, conspiracy, and hidden systems, suggesting an emergent "Foucault’s Pendulum mindset" driven by the underlying corpus and connection-finding task.
Technical Components: Key tools included selectolax for EPUB parsing, SQLite and sqlite-vec for data storage and embeddings, wtpsplit for sentence segmentation, DSPy for structured LLM calls, and specific models (google/embeddinggemma-300m, BAAI/bge-reranker-v2-m3) for embedding and reranking.
External Critique: Commenters in the associated discussion thread expressed skepticism regarding the philosophical depth and validity of the LLM-generated connections, describing some links as trivial or arbitrary ("The actual data or outputs seem to be more or less...trash?"). Counterarguments defended the technical achievement and the utility of LLM-mediated reading as a starting point for human thought.