This presentation details Reddit's journey in enhancing service discovery for its vast, multi-region Kubernetes infrastructure. Faced with the limitations of Kubernetes' default DNS-based service discovery at their immense scale (tens of billions of requests daily) and with the "Worldwide Reddit" initiative, they developed a sophisticated proxyless gRPC XDS control plane.
The solution addresses the complexity for developers and the need for dynamic routing by introducing several novel components: a Config Map Injector to dynamically provide bootstrap.json configurations to clients, allowing for flexible XDS server addressing and bulkhead creation; and a Validating Webhook that not only improves user experience by rejecting invalid gRPC routes but critically protects the control plane from "queries of death" by pre-validating configurations. The system is bolstered by extensive client-side observability through gRPC XDS metrics, custom tracing, and a Client Status Discovery Service (CSDS) endpoint, providing crucial insights into opaque network paths and client configurations. A Sampling Client facilitates progressive and easily reversible migration to the new XDS-based system, with a discussion on the trade-offs between per-RPC and per-connection sampling.
Reddit's Advanced Service Discovery with XDS
0:00 Reddit's Scale and Tech Stack: Reddit processes tens of billions of requests daily, leveraging Kubernetes, gRPC, Python, and Go for its infrastructure.
0:46 Kubernetes Default Service Discovery: Kubernetes provides basic service discovery via DNS (e.g., service.namespace.svc.cluster.local), returning either a cluster IP or a list of pod IPs.
1:53 Scaling Challenges with Default DNS: The default DNS-based service discovery model became insufficient for Reddit's "Worldwide Reddit" project, which involves spinning up multiple clusters and regions, leading to complex decision-making for service owners and difficulties with infrastructure evolution.
2:50 Proxyless Approach Decision: Reddit chose a proxyless approach for its gRPC services over a proxy-based model due to higher isolation, lower overhead, and more intuitive operational costs, despite proxies offering broader protocol support.
5:37 The XDS Control Plane: The core solution is a proxyless gRPC XDS control plane, implemented as a Kubernetes controller, which reconciles gRPC routes, endpoint slices, and services to generate and advertise XDS snapshots to clients.
6:32 Config Map Injector for Dynamic Bootstrap: To dynamically manage the bootstrap.json file required by gRPC XDS clients, a Config Map Injector controller watches Kubernetes namespaces and creates config maps containing the bootstrap information, allowing dynamic changes (e.g., for bulkheads or penalty box control planes) without client modifications.
8:09 Validating Webhook for Control Plane Protection: A validating webhook is used not only for improved user experience by rejecting invalid resources but also to protect the control plane from "queries of death" by pre-validating gRPC routes, preventing potential crashes.
9:22 Enhanced Observability: Reddit implemented client-side gRPC XDS metrics, custom metrics with tracing, and a Client Status Discovery Service (CSDS) endpoint. These tools provide deep insights into the data path and client configurations, essential for debugging in an increasingly opaque network environment.
10:47 Sampling Client for Progressive Migration: A sampling client was developed to enable a progressive and easily revertible migration to the new XDS-based system, allowing RPC calls to selectively use either XDS-provided addresses or the default DNS resolver.
12:10 Sampling Trade-offs: Sampling can be done "per RPC" (instant rollback but high connection overhead due to managing two dials) or "per connection" (lower overhead but may require pod restarts for rollbacks).
13:10 CSDS Usefulness: The CSDS endpoint has proven useful for debugging misconfigurations and understanding a client's exact configured view of the world, serving as the "ultimate source of truth" for asynchronous problem-solving.
14:03 State-of-the-World Updates: gRPC clients for XDS primarily support "state-of-the-world" updates (full snapshots) rather than incremental delta updates. Reddit's control plane optimizes this by creating separate snapshots for rapidly changing endpoints versus more stable routes and clusters.
15:39 Update Batching to Reduce Churn: The control plane implements a coalescing window (5-10 seconds) to batch endpoint updates. This strategy reduces churn and prevents clients from being overwhelmed with constant messages during periods of high change, aligning with their previous DNS caching approach.