This discussion introduces AWS Aurora Distributed SQL (dSQL), a new offering announced at re:Invent, designed to redefine database scalability by "defying physics." Kate Gavon explains dSQL's core innovation: the creation of isolated "micro VMs" for each transaction. This unique approach, fundamentally different from competitors like Cloud Spanner, eliminates traditional database locks and enables theoretically limitless connections and high concurrency.
While dSQL offers unprecedented transaction scalability, it employs optimistic locking, meaning transactions may fail at commit if conflicting changes occurred, necessitating application-side retry logic. A crucial limitation is a strict 5-minute transaction timeout, making dSQL ideal for Online Transaction Processing (OLTP) workloads with many short, fast transactions (e.g., online gaming, banking) but unsuitable for long-running analytical queries. It also features a PostgreSQL-compliant core but with certain traditional relational features (like foreign keys and stored procedures) removed to facilitate its distributed nature, pushing referential integrity concerns to the application layer. The release is a bare-bones MVP, with future feature expansions like JSON and PG Vector support anticipated, reflecting AWS's strategy of evolving products based on real-world customer feedback.
Exploring AWS Aurora Distributed SQL: A Deep Dive into its Scalability and Implications
0:00 A Game-Changing Announcement: AWS introduces Aurora Distributed SQL (dSQL), a new offering described as "defying physics" in its approach to database scalability, competing with Cloud Spanner and CockroachDB.
0:58 Revolutionary Transaction Handling: dSQL's primary innovation is creating isolated "micro VMs" (akin to Lambda functions) for each transaction, eliminating traditional database locks and allowing for virtually unlimited concurrent connections and writes.
3:00 Lock-Free Operations: Each transaction operates in its own isolated VM, fetching data consistent with its timestamp using PostgreSQL's Multi-Version Concurrency Control (MVCC) system, and only committing changes to the main storage at the very end.
4:48 dSQL vs. Aurora Limitless: dSQL is optimized for extreme concurrency (many connections, small transactions), whereas Aurora Limitless focuses on storage scalability through sharding. There are currently no plans to combine these two distinct offerings.
5:57 Optimistic Locking Model: Unlike traditional databases that use pessimistic locking (locking rows at transaction start), dSQL uses optimistic locking, where conflicts are only detected at the commit stage. This enhances concurrency but requires applications to implement retry logic for potential transaction failures.
9:40 Transaction Timeout and Workload Suitability: dSQL imposes a strict 5-minute hard timeout for transactions. This makes it unsuitable for long-running analytical (OLAP) queries, firmly positioning it as an Online Transaction Processing (OLTP) database for fast, numerous, and typically simple transactions.
11:36 Ideal Use Cases: Recommended for applications with high concurrency and frequent, small updates that rarely conflict on the same data rows, such as online gaming leaderboards, banking systems, and e-commerce platforms.
13:13 Application-Level Responsibility: Due to optimistic locking, applications using dSQL must incorporate robust retry mechanisms and handle referential integrity (as foreign keys are not supported).
14:00 Global Time Service: The system relies on an incredibly accurate and globally synchronized time service to ensure consistency across active-active multi-region deployments.
16:08 Reduced PostgreSQL Feature Set: To achieve its distributed scale, dSQL currently does not support PostgreSQL features like foreign keys, stored procedures, or JSON data types (though JSON support is expected soon). It is described as a "SQL light" or "cut-down" version of PostgreSQL.
17:47 PostgreSQL-Only Focus: There is no current expectation for a MySQL version of dSQL, as PostgreSQL's MVCC and open-source nature make it the preferred foundation.
22:25 Migration Considerations: Existing Aurora users migrating to dSQL will likely need to use services like DMS (Database Migration Service) rather than a one-click migration, due to the fundamental architectural and feature differences.
27:49 AWS MVP Release Strategy: dSQL's initial release is a "bare bones" Minimum Viable Product (MVP), typical for AWS. Significant feature additions and improvements are anticipated in the coming months based on direct customer feedback.
29:53 Current Missing Features: As of launch, dSQL lacks support for PG Vector (for AI embeddings), CloudFormation, and Terraform, which are common initial omissions for new AWS services.