This episode of the Bug Bash podcast features the three co-founders of FoundationDB discussing the inception, development, and market journey of their distributed transactional database. They recount how the idea originated from a frustration with existing database solutions' inability to offer both scalability and strong data consistency (ACID transactions). The core innovation was a layered architecture, building a robust, distributed transactional key-value storage engine designed to underpin various higher-level data models. Facing initial market skepticism that their goal was "impossible," they leveraged innovative deterministic simulation testing and even a physical fault-injection setup to validate their system's extreme reliability. This rigorous testing, which consumed a significant portion of their engineering effort, became their competitive advantage, enabling rapid development, high confidence in releases (even without pre-commit code review), and ultimately leading to acquisition by Apple and subsequent open-sourcing. The founders reflect on the enduring value of FoundationDB's reliability for companies building complex, custom data layers, many of which remain proprietary.
The FoundationDB Journey: From "Impossible" Idea to Unprecedented Reliability
0:00 Founding Vision: The co-founders (Dave Rosenthal, Dave Sharer, Nick Levetszo) discuss starting FoundationDB to address the limitations of existing databases (outdated relational, NoSQL sacrificing consistency for scale) when trying to build scalable services.
3:06 Debunking Dogma: They challenged the conventional wisdom, based on the CAP theorem, that achieving both high availability and ACID transactions in a distributed database was impossible.
3:50 Layered Architecture: FoundationDB was designed as a "storage engine" – a distributed transactional key-value store – forming a reliable base upon which various data models (e.g., relational, queues, search engines) could be built. This novel approach provided ACID guarantees across multiple machines.
7:00 Market Challenges: Initially, their product's appeal was niche, targeting companies willing to build their own high-level databases (like Snowflake). This resulted in a very long sales cycle and slow revenue growth.
8:02 Overcoming Skepticism: The market's initial reaction was disbelief ("This is not possible"). They famously demonstrated the database's durability at TechCrunch Disrupt using a physical setup that allowed attendees to unplug and shut down computers, proving its resilience and generating significant media attention.
14:42 Obsession with Testing: Recognizing the criticality of correctness in a distributed system, they committed heavily to testing to avoid production bugs and data corruption.
16:05 Deterministic Simulation: A radical approach involved running the real database software within a custom, single-threaded, deterministic simulator. This required significant architectural constraints (e.g., multi-process, single-threaded model, custom C++ dialect).
18:38 Doubling Down on QA: An early severe bug, undiscovered by their initial tests, led them to significantly increase investment in their simulation and testing framework, seeing it as the only path to achieve the required level of perfection.
21:05 Physical Validation: A physical fault tolerance testing setup with electronically controlled power switches was built to validate the simulation, finding only two bugs: one in their Apache Zookeeper dependency (prompting them to replace it with their own Paxos implementation built in weeks) and an installation durability issue.
25:50 Early Roots in Determinism: The co-founders' prior experience building a multiplayer video game at age 14 taught them the value of determinism for debugging complex, synchronized systems, providing confidence for FoundationDB's design.
27:50 Apple Acquisition & Open Source: FoundationDB was acquired by Apple and later open-sourced, largely due to internal efforts by Ben Collins, making its powerful technology accessible.
29:19 Lasting Impact & Reliability: FoundationDB gained a "sterling reputation" for solving the hardest problems in distributed systems (reliability, scalability, ACID transactions). Many companies have built valuable, high-reliability custom data layers on top, although most remain proprietary.
31:50 Unprecedented Development Velocity: The deterministic simulation testing was indispensable, allowing the team to maintain rapid development velocity (e.g., a 14x speed increase in transaction processing with a small team) and confidently ship code, even without pre-commit code review, because of the rigorous, high-confidence testing process.
38:13 Advanced Testing Techniques: They developed sophisticated testing techniques like "swizzle clogging," which randomly induced network partitions and power failures, simulating billions of hours of real-world failure scenarios in a single night.