This presentation addresses the fundamental challenge faced by programmers: the impossibility of objectively capturing complex reality within discrete computer systems. Speaker Lily, an information security engineer and ex-historian, argues that many project failures stem from the false belief that reality is simple and easily quantifiable. The talk elaborates on four main fallacies: that more data equates to perfect representation (ignoring that "raw data" is an oxymoron), that everything important is measurable (leading to reliance on imperfect proxies), that reality is uniform across all observers, and that the world possesses inherent structure (discussing how arbitrary categorization flattens complexity). Using examples from LLMs, historical forestry, and algorithmic aesthetic trends ("Airspace"), the speaker highlights the dangers of mistaking models for objective truth. The talk concludes by advocating for "mindful abstraction": treating abstractions as useful tools built for specific purposes with explicit constraints, rather than striving for unattainable perfect mirrors of reality.
Falsehoods Programmers Believe About Reality
0:30 Reality vs. Abstraction: The core issue is that reality cannot be captured objectively, yet computers require simplified, abstract representations (models and data sets) of the world to function.
2:16 Talk Purpose: The goal is to provide tools for developers and stakeholders to challenge the premise that reality can be captured objectively, thereby avoiding brittle, poorly designed systems.
7:00 Fallacy 1: Quantity Over Quality: The belief that "more data" (e.g., 10 billion sources) leads to a perfect representation of reality is false. All data is processed and biased, making "raw data" an oxymoron resulting from human choices about measurement.
8:51 LLM Example: The widespread use of LLMs trained on massive, low-quality data (like years of unorganized corporate documents) often results in non-useful output because quantity was prioritized over quality and purpose.
11:21 Bias in Language: LLM training data is primarily scraped written text, representing only a small, biased slice of human language (which also includes gestures, tone, and interaction over millennia) and favoring groups with early publication access.
13:39 Fallacy 2: Everything is Measurable: Concepts like security, justice, or love are not directly measurable. Systems rely on proxies (e.g., vulnerability patch counts for security), leading to flawed decision-making if the proxy is mistaken for the true measure of the concept.
14:23 Fallacy 3: One Single Reality: Observers and describers bring inherent biases and unique experiences (including physical perception, history, and opinion). While consensus is necessary for social function (e.g., web standards), striving for a single objective view is misleading. Bias can only be managed, not removed, through evolving consensus.
17:45 Fallacy 4: Reality Has Inherent Structure: Reality has no inherent structure; all categories (e.g., is a hot dog a sandwich?) are artificial and imposed. When these artificial categories are enforced by powerful entities (governments/companies), they flatten local knowledge and can lead to negative consequences.
20:42 Forestry Example (Categorization Impact): The historical attempt at "scientific forestry" in Prussia/Saxony, which replaced diverse forests with legible, single-species rows for resource prediction, led to ecological collapse once the complexity sustaining the soil health was removed.
24:26 The Main Trap & Solution: Striving for a perfectly complete capture of reality is paralyzing and useless. Systems should instead be built as "tools, not mirrors" (25:34), prioritizing utility over truth for a specific, well-articulated purpose.
27:00 Abstraction Principles: Mindful abstraction requires designing with clear context and purpose (being explicit about constraints) and acknowledging the ethical consequences of the design choices (e.g., who benefits, who is excluded).