This presentation by Rahul from LinkedIn discusses advanced techniques for tuning gRPC services to achieve high performance, low latency, high throughput, and stability at scale in production environments. The talk focuses on three key areas: deadlines, batching, and keep-alives.
For deadlines, the speaker explains their cascading nature in distributed systems, where time budgets are consumed at each service hop, potentially leaving downstream services with insufficient time. He highlights issues like miscoordination with retries, precision loss with the relative grpc-timeout header, and amplification caused by aggressive retries or hedging. Solutions include using absolute deadlines, clamping relative timeouts, and setting meaningful per-attempt budgets with caps on retries.
Regarding batching, the benefits for both CPU-bound workloads (e.g., ML inference) and network efficiency are detailed. Batching multiple requests into a single RPC reduces overhead from serialization, object allocation, and network chatter, leading to improved throughput and lower tail latencies. Strategies for implementing batching, such as size-based or time-based flushing, are also discussed.
Finally, keep-alives are presented as crucial for maintaining stable connections, preventing idle connection drops by NAT devices and load balancers, and acting as an early warning system for network path issues. The talk covers best practices for tuning keep-alive intervals to align with network infrastructure timeouts, introducing jitter to avoid "thundering herd" scenarios, and implementing server-side guardrails against abusive or misconfigured client behavior, including auto-tuning for dynamic network conditions.
Optimizing gRPC for High Performance: Deadlines, Batching, and Keep-Alives
0:00:02 - Introduction & Core Topics: Rahul, a staff engineer at LinkedIn, presents on tuning gRPC for high performance, focusing on three key areas: deadlines, batching, and keep-alives. The session is aimed at engineers already familiar with gRPC in production who want to achieve low latency, high throughput, and stability at scale.
0:01:02 - Challenges with gRPC at Scale: While gRPC offers strong typing, efficient Protobuf serialization, and streaming, large-scale production deployments face issues like increasing latency, uncoordinated retries, connection churn, and complexities with load balancers and proxies. Tuning these aspects can significantly improve system performance.
0:03:02 - Deadlines and Downstream Impact: Deadlines act as a request budget, specifying how long a client waits for a response. They travel downstream, meaning each subsequent service receives a shorter remaining time. Deep service stacks can leave the furthest downstream service with inadequate time to perform its work, leading to context deadline exceeded errors.
0:04:28 - Implementing Fail-Fast with Deadlines: Servers should check the remaining deadline and "fail fast" if there's insufficient time (e.g., <50ms) to complete meaningful work. This prevents resource waste and enforces predictability, even if it results in a deadline exceeded error.
0:05:23 - Deadline-Retry Coordination: gRPC won't retry a call after its deadline expires. Misconfigurations, such as short deadlines with many retries or long deadlines causing excessive retries, can lead to wasted resources and increased load, especially when upstream services have longer deadlines than downstream ones.
0:06:59 - Mitigating grpc-timeout Precision Loss: The grpc-timeout header transmits a relative remaining time, not an absolute timestamp. This can lead to cumulative precision loss (rounding errors) across multiple proxies and sidecars, reducing the actual time available to backend services.
0:08:46 - Solution for Precision Loss: To prevent precision loss, carry an absolute deadline within the context. Each service hop should recompute the relative grpc-timeout header from this absolute deadline and clamp it to a safe minimum value to avoid it rounding down to zero.
Retry Amplification: A long end-to-end deadline combined with very short per-attempt timeouts can cause clients to make many rapid, failing retries, wasting CPU and network resources.
Hedge Amplification: Parallel hedging (sending multiple copies of a request) can flood downstream services if the remaining deadline budget is too short for the hedging delay and roundtrip time.
0:10:52 - Strategies to Prevent Amplification: Set meaningful per-attempt time budgets for requests, cap the maximum number of retry attempts, and for hedging, only enable it if the remaining deadline is greater than the expected roundtrip latency.
0:11:52 - Benefits of Batching Requests: Batching involves bundling multiple client-side requests into a single RPC. This approach pays the CPU cost (Protobuf encode/decode, object allocation, header parsing) and network overhead (round trips, headers) only once per batch, significantly improving efficiency, reducing congestion, and lowering tail latencies.
0:13:33 - When Batching is Most Effective: Batching is particularly impactful for CPU-bound workloads like ML inference (on CPUs or GPUs, enabling vectorized operations) and streaming traffic, which often arrives in bursts.
0:14:57 - Batching Implementation Strategies: Batching can be implemented using either a size-based approach (flushing when a buffer reaches a certain number of items, e.g., 16 or 64) or a time-based approach (flushing after a specific time window, e.g., 10 milliseconds).
0:16:52 - Purpose of Keep-Alives: Keep-alives serve as an early warning system for connection issues (e.g., Wi-Fi flips, mobile sleep) and are crucial for maintaining active connections by keeping NAT and load balancer mappings warm. They prevent idle connections from being silently dropped due to timeouts.
0:17:48 - How Keep-Alives Work: Clients send small HTTP/2 ping frames (e.g., every 45 seconds) and expect an acknowledgment. This activity is recognized by load balancers and NAT devices as real traffic, preventing their idle connection timers from expiring.
0:18:18 - Tuning Keep-Alive Settings: Keep-alive intervals should be tuned to be 70-80% of the shortest idle timeout in the network path (e.g., 45-50 seconds if network drops after 60 seconds). Adding jitter (e.g., +/- 15-20%) to ping times prevents a "thundering herd" problem from many clients pinging simultaneously.
0:19:31 - gRPC HTTP/2 Ping vs. OS TCP Keep-Alives: Prefer gRPC's built-in HTTP/2 ping as it's lighter-weight and more precise. If both are used, ensure they are not set at the same cadence to avoid interference and double the network noise.
0:20:03 - Server-Side Keep-Alive Guardrails: Servers should enforce policies for keep-alives, such as a minimum ping interval (to prevent abuse/misconfiguration) and disabling permit_without_stream by default (preventing pings on completely idle connections, except for specific long-lived mobile/control plane scenarios). Violations should result in a GOAWAY signal.
0:21:07 - Auto-Tuning Keep-Alives: In dynamic network environments (e.g., a phone switching networks), keep-alive intervals can be dynamically shortened to quickly revalidate a connection path if it's suspected to be broken, preventing prolonged RPC failures.