Persona: Senior Cloud Architecture & Systems Reliability Engineer (SRE)
Abstract:
This discussion features Milon, VP of Data and Analytics at AWS, detailing the immense scale, engineering complexities, and architectural evolution of Amazon S3. The conversation emphasizes the sheer magnitude of the service, currently storing over 500 trillion objects and hundreds of exabytes of data, handling hundreds of millions of transactions per second. Key engineering topics covered include the foundational shift from eventual consistency to strong consistency—achieved via a proprietary replicated journal and cache coherency protocol without incurring latency or cost penalties—and the engineering discipline required to manage failure domains (correlated failure, crash consistency, failure allowances) at this scale. Furthermore, the evolution of S3 beyond unstructured object storage is explored, highlighting the introduction of native structured data primitives like S3 Tables (built on Apache Iceberg) and the recently launched S3 Vectors for semantic understanding via AI embeddings. The underlying engineering philosophy centers on maintaining core S3 tenets (durability, availability) while leveraging technical fearlessness to continuously innovate and simplify the user model, reinforced by the rigorous application of formal methods (automated reasoning) to verify correctness.
Reviewing S3 Architecture and Scale: Insights for Systems Engineers and Data Architects
0:00:08 Scale Metrics: S3 currently holds over 500 trillion objects, hundreds of exabytes of data, processes over a quadrillion requests annually, and serves hundreds of millions of transactions per second. The underlying infrastructure includes tens of millions of hard drives across millions of servers in 120 Availability Zones (AZs) across 38 Regions.
0:04:15 S3 Origins & Initial Consistency Model: Launched in 2006, the initial design was anchored around eventual consistency to optimize for durability and availability, suitable for early e-commerce use cases where temporary data listing delays were acceptable.
0:06:41 Evolution to Data Lakes: The adoption of tools like Hadoop drove the use of S3 for unstructured data, eventually leading customers to store structured data (e.g., Parquet files) in what became known as "data lakes," utilizing formats like Apache Iceberg.
0:08:02 S3 Primitives: The fundamental operations remain PUT and GET, supplemented by newer native primitives: S3 Tables (managing structured data via Iceberg compliance) and S3 Vectors (a new data structure for storing embeddings).
0:11:04 Conditionals and Evolution: Recent additions include conditional operations like PUT if absent and DELETE if match, demonstrating continuous refinement based on application behaviors.
0:14:34 Pricing Philosophy: The mission is to provide the best storage service, achieved partly by continuously lowering costs (storage rates have dropped significantly since the 15 cents/GB launch price) to ensure data growth remains economically viable for customers, utilizing features like Intelligent Tiering.
0:17:55 Glacier Architecture: Extreme cost reduction (e.g., 1 cent/GB for Glacier) is achieved by deep engineering efficiencies across the entire stack, from hardware layout to data center operations, managing deep constraints on availability and cost.
0:20:35 Transition to Strong Consistency: The system evolved past eventual consistency by implementing a replicated journal (a distributed data structure chaining nodes sequentially) combined with a cache coherency protocol to ensure the index subsystem guarantees the most recent PUT is reflected in subsequent reads.
0:26:50 Trade-offs Absorbed: AWS made an explicit decision to implement strong consistency—including the required engineering overhead (replicated journal and cache coherency)—without increasing latency or charging customers for the feature.
0:29:03 Correctness via Formal Methods: To verify the complex strong consistency model at scale, S3 employs automated reasoning (formal methods) and proofs that are incorporated into check-ins for the indexing subsystem to prevent regressions.
0:36:36 Durability Assurance (11 Nines): Durability is verified through a fleet of auditor microservices that inspect every byte, trigger repair systems when needed, and continuously report on adherence to the durability promise, treating component failure as an expected, constant event.
0:40:27 Correlated Failure: A critical design consideration is preventing correlated failures (where multiple components fail simultaneously due to a single fault domain, e.g., a single rack or AZ). Replication across many AZs directly mitigates this risk for availability.
0:42:25 Crash Consistency: Systems are designed to always return to a consistent state after any fail-stop failure, a key part of the engineering mindset.
0:59:59 Engineering Tenet: Scale is Advantage: New features, such as S3 Vectors, are designed such that increasing scale improves performance (e.g., workload decorrelation), rather than degrading it.
0:58:00 S3 Vectors Implementation: Vectors (embeddings) are a new primitive utilizing vector neighborhoods computed offline and asynchronously. Queries locate the nearest neighborhoods, load relevant vectors into fast memory, and apply the nearest neighbor algorithm, achieving sub-100ms performance for warm queries against up to 20 trillion vectors.
1:09:54 Simplicity as a Core Value: Despite internal complexity, S3 maintains simplicity in its user model (simple API, SQL access, easy vector understanding via AI).
1:11:42 Recommended Trait for Engineers: Relentless curiosity and the willingness to redefine boundaries ("draw new lines") rather than simply adhering to existing architectural constraints.