This presentation outlines Reddit's approach to modernizing its service discovery infrastructure to support its massive scale and global expansion, "Worldwide Reddit." Initially relying on Kubernetes' native DNS-based service discovery, Reddit encountered significant challenges with complexity, manual configuration, and varied load balancing requirements across multiple clusters and regions. To overcome this, Reddit adopted a proxyless gRPC XDS-based system. The solution features a control plane comprising an XDS Kubernetes controller for configuration snapshot generation, a Config Map Injector for dynamic client bootstrap configuration, and a Validating Webhook to ensure configuration validity and protect the control plane from malformed inputs. To facilitate gradual adoption and maintain robust operations, Reddit implemented extensive client-side observability tools, including gRPC XDS metrics, custom tracing, and a Client-Side Discovery Service (CSDS) endpoint. A flexible sampling client was also developed, allowing for progressive and reversible migration of RPC traffic using either per-RPC or per-connection strategies, balancing control with operational overhead.
Reddit's Scalable Service Discovery with gRPC XDS
0:00 Reddit's Scale & Technology Stack: Reddit is a massive platform, leveraging Kubernetes to run all its services, extensively using gRPC, Python, and Go, and handling tens of billions of requests daily.
0:46 Kubernetes Native Service Discovery: Out-of-the-box Kubernetes uses pods (compute units), deployments, and services for networking. It offers DNS resolvers providing addresses like service.namespace.service.cluster.local, which return either a cluster IP or a list of pod IPs.
1:54 Challenges with Scale & Multi-Region Deployment: As Reddit scaled globally with its "Worldwide Reddit" project, Kubernetes' basic DNS-based service discovery became inadequate. Service owners faced complex decisions regarding local vs. remote service calls and manual management of client/server details.
2:50 Need for Advanced Load Balancing: Services had diverse load balancing requirements, which were cumbersome to manage through client-side changes, contributing to developer confusion.
3:25 Proxy vs. Proxyless Decision: Reddit weighed using a proxy (like Envoy) versus a proxyless approach. They chose a proxyless solution, primarily due to:
gRPC Focus: gRPC was their main protocol.
High Isolation: Better application isolation, as code runs within the service.
Intuitive Operations: Application developers found debugging logs and metrics within their own service more straightforward.
5:37 Control Plane Components: Reddit built a custom control plane with three key components:
5:50 XDS Control Plane: A standard Kubernetes controller that reconciles gRPC routes, endpoint slices, and services to generate and advertise XDS snapshots to clients.
6:32 Config Map Injector: This novel controller watches Kubernetes namespaces and dynamically creates a bootstrap.json ConfigMap for gRPC XDS clients. This allows changing the XDS server address (e.g., for bulkheads or penalty boxes) without requiring client code modifications, empowering the platform team and ensuring static failure to known good configurations.
8:09 Validating Webhook: Beyond improving user experience by rejecting invalid resource creations, this webhook critically protects the control plane from "queries of death" (malformed gRPC routes that could crash the XDS snapshot generation process) by validating configurations before they are processed.
9:22 Adoption & Observability: To ease adoption and debugging, Reddit focused on:
9:31 Client-Side Observability: Exposing gRPC XDS client metrics and using custom metrics with tracing to understand the data path, especially as networking becomes more opaque to developers.
10:24 CSDS Endpoint: Provides an administrative interface to query any client for its current XDS configuration, serving as an ultimate source of truth for troubleshooting misconfigurations.
10:48 Sampling Client: A crucial tool for progressive and reversible migration, allowing RPC calls to selectively use either the new XDS addresses or the default DNS resolvers.
11:10 Sampling Trade-offs (Per-RPC vs. Per-Connection):
Per-RPC Sampling (11:22): Offers instant rollback and fine-grained control but introduces higher overhead due to managing two sets of connections (XDS and DNS).
Per-Connection Sampling (12:11): Lower overhead, as the dial type (XDS or DNS) is chosen once per connection, but may require pod restarts for rollback, reducing agility.
14:03 XDS Update Mechanism: Reddit's XDS implementation uses a "state of the world" approach (not incremental updates), which gRPC clients currently support.
14:46 Efficient Snapshotting & Coalescing: The control plane separates snapshots for fast-changing endpoints and more stable routes/clusters. It also employs a 5-6 second coalescing window to batch endpoint updates, reducing churn and preventing clients from being overwhelmed during periods of high activity.