This presentation from Netflix outlines their comprehensive approach to handling severe traffic spikes through "automated prioritized load shedding" without increasing server capacity. Benjamin Fedorca from the Java platform team details how they evolved from basic load shedding to a sophisticated system that prioritizes requests based on criticality, leveraging detailed per-RPC latency measurements and desharding strategies. The system employs gRPC interceptors and Envoy filters to enforce these priorities across the call stack, ensuring essential services remain functional during extreme load. A key innovation is an automated platform that continuously generates, tests, validates, and deploys resilience configurations, eliminating the need for manual tuning and enabling rapid adaptation to changing traffic patterns.
Handling Traffic Spikes with Automated Prioritized Load Shedding at Netflix
0:02 The Core Challenge: Benjamin Fedorca of Netflix's Java platform team introduces the goal of surviving significant traffic spikes without increasing server capacity, a complex problem given the unique performance profiles of hundreds of microservices.
2:41 Why Not Just Autoscale? While Netflix utilizes pre-scaling and autoscaling, these methods are costly, can hit capacity limits, and are often too slow to react effectively to sudden, intense load spikes, potentially leading to cold start performance issues.
4:01 The Danger of Congestive Failure: Without proper handling, systems experiencing load spikes can rapidly transition to "congestive failure," where utilization hits 100%, nearly all requests fail, and the system struggles to recover, often requiring costly restarts.
5:23 Introducing Success and Failure Buffers: Netflix employs a "success buffer" (reserved capacity for normal operations and small spikes) and a "failure buffer" (reserved capacity to gracefully reject excess requests, preventing catastrophic failure and allowing for graceful degradation).
6:17 Granular System Utilization Measurement: Instead of broad estimates, system capacity is now measured by objectively tracking concrete signals like CPU utilization, resource queuing, and crucially, individual RPC latency distributions. Each server continuously analyzes its own performance to determine system utilization and shed requests when too high.
7:39 Prioritized Load Shedding: Recognizing that not all requests are equally critical (e.g., pre-fetch data vs. playing content), Netflix implemented a priority-aware traffic management system. As load increases, lower-priority requests are progressively shed to ensure the most critical traffic continues to be served, even in extreme congestion.
8:35 Stack-Wide Application: This priority-aware logic is applied at all layers of the service stack, enabling more responsive degradation of specific capabilities deep within the call graph rather than just at the user-facing edge.
9:29 Desharding for Enhanced Critical Capacity: Previously sharded traffic by criticality for cost savings, Netflix now deshares clusters. This consolidates resources, dramatically increasing the "success buffer" for critical traffic and improving overall resource utilization during both normal and peak loads.
10:25 Benefits of the Combined Approach: The integrated solution (refined load shedding, prioritized requests, desharding) provides necessary capacity for critical requests, ensures baseline traffic continuity, enables automated and rapid responses, and allows for graceful degradation of specific functionalities.
11:40 Automated Configuration and Validation Platform: Due to the extensive tuning required for each RPC and cluster, Netflix developed an automated platform that continuously analyzes historical data, generates proposed load shedding/autoscaling configurations, tests them with simulated traffic spikes, validates the results, and automatically applies them to production clusters.
14:04 Technical Implementation Details: The system is primarily implemented using gRPC client interceptors (to annotate request priority), Envoy proxy sidecar filters (to make shedding decisions), and gRPC service interceptors (to monitor latency distributions and compute system utilization).
15:35 Key Load Shedding Signals (Q&A): Critical signals include CPU utilization, CPU queuing (load factor), and latency distribution (especially tracking P90 or P99 latencies, and if P20 crosses P99, indicating severe load). It's crucial to track these per-RPC to handle multimodal latencies effectively.
18:31 Managing Feedback Loops (Q&A): The system mitigates potential feedback loop issues by intentionally measuring latency distributions during known "good" (non-load shedding) periods, ensuring the system learns from stable states. Configurations are automatically re-tuned every few weeks or as needed.
19:52 Static vs. Dynamic Configuration (Q&A): Netflix favors pre-baked, validated resilience configurations over fully dynamic client-side reactions, as dynamic clients can struggle to learn from ambiguous (good or bad) system states, leading to operational difficulties.