This presentation by Google software engineers Yash and Abhishek provides a comprehensive overview of gRPC observability, highlighting recent advancements, particularly with Open Telemetry, and a suite of complementary diagnostic tools.
The core of the update focuses on gRPC's enhanced integration with Open Telemetry for tracing and metrics. This includes the development of gRPC-specific semantic conventions to capture nuances beyond generic RPC systems, the implementation of Open Telemetry tracing in Java, C++, and Go (currently experimental), and the introduction of detailed TCP traces in C++ for network-level debugging. On the metrics front, the presentation clarifies the distinction between "per attempt" and "per call" client metrics, and introduces new metrics for retries/hedges, WR/XDS load balancing, and improved subchannel visibility, including disconnection error causes. Future metric plans include deeper TCP-level insights.
Beyond Open Telemetry, the speakers discuss established and community tools: gRPC Binary Logging for troubleshooting, load testing, and RPC replay; gRPCurl for command-line interaction and API exploration; ChannelZ for inspecting channel/socket states with a helper UI; and gRPC Debug for querying gRPC processes and XDS configurations. The roadmap emphasizes stabilizing tracing, rolling out new metrics across languages, and improving latency profiling tools. The presentation also briefly addresses the ongoing challenge of providing comprehensive observability for bidirectional streaming RPCs.
gRPC Observability: Latest Updates and Tools
0:00 Introduction to gRPC Observability: Yash and Abhishek from Google discuss recent advancements in gRPC observability, with a focus on Open Telemetry and other ecosystem tools.
0:48 Open Telemetry Integration: gRPC has integrated with Open Telemetry, an open-source observability framework. Users can set up plugins with minor code changes to export deep gRPC data to any Open Telemetry-supported backend, including Google Cloud.
1:35 gRPC-Specific Semantic Conventions: Due to the generic nature of Open Telemetry's RPC semantic conventions, gRPC has defined its own, more nuanced metrics and traces through GRFCs (gRPC's version of RFCs), with ongoing collaboration to influence broader Open Telemetry RPC standards.
2:47 Open Telemetry Tracing (Experimental): New tracing capabilities provide detailed, tree-like structures of spans for sampled requests in distributed systems, showing the timeline of requests across services. This feature is implemented in Java, C++, and Go but is currently experimental, awaiting cross-language interoperability tests for stability.
4:00 TCP Traces in C++: gRPC now instruments TCP traces in C++ (Linux kernels only), providing events for when TCP packets are passed to the kernel, scheduled, sent, and acknowledged. This offers stats like delivery rate and retransmission data, crucial for debugging network-related latencies within Google.
5:28 The Importance of Metrics: Metrics are vital for understanding the health and performance of gRPC applications, offering visibility to monitor, optimize, pinpoint bottlenecks, and act as an early warning system for issues like latency and error rates.
Retries and Hedges: Functionality from OpenCensus is brought over to track retries, hedges, and the delays between them.
WR and XDS Metrics: Implemented in core and Java/Go (WR) and core/Java (XDS).
Subchannel Metrics (Improved Visibility): Replaces "pick first" metrics to provide clearer connection visibility, including the actual disconnection errors (e.g., socket errors, connection timeouts), improving debugging.
Outlier Detection Metrics: An important community contribution from Dropbox for identifying performance anomalies.
Optional gcp_backend_service Label: Allows easy slicing and dicing of metrics for clients talking to multiple backend services.
10:53 Future of Metrics: TCP Level: A proposal is underway for new TCP-level metrics to provide deeper insight into transport layer activity, including TCP min RTT, delivery rate, packet send, retransmitted, and spurious retransmissions, to diagnose intermittent network issues.
12:32 gRPC Binary Logging: A feature to record RPCs in a binary format for troubleshooting (requests, responses, statuses), load testing (analyzing real traffic), and replaying RPCs in dev environments to reproduce bugs. Configuration allows specific logging of services/methods and message size limits. Security is maintained by separating metadata from payload, enabling filtering of sensitive data.
14:20 gRPCurl (Community Tool): A command-line tool for directly interacting with gRPC servers, enabling sending requests and getting responses from the terminal. It speeds up testing/debugging, facilitates API exploration (with server reflection), and is useful for scripting automated tests or health checks.
15:49 ChannelZ (Admin Service): An admin service that provides information on channels, subchannels, servers, and sockets, offering insights into channel state, recent events, and subchannel issues. A helper UI tool in the gRPC ecosystem fetches and presents this data. A ChannelZ V2 proposal aims for a more generalized structure.
17:21 gRPC Debug (CLI Utility): A command-line utility to query gRPC processes, acting as a gRPC client. It supports querying ChannelZ (as an alternative to the UI), health checks (via the health check service), and XDS (via the client status discovery service) to check resource status and configuration.
Additional Metrics: Implement TCP-level metrics and roll out new subchannel and outlier detection metrics to more languages (C++, Go, Java).
Tracing Stabilization: Stabilize Open Telemetry tracing with interoperability tests.
Latency Profiling: Improve user-friendly data extraction from an existing C++ latency profiling tool for visualization with tools like Perfetto.
19:48 Q&A: Bidirectional Streaming Observability: The current focus has been on unary RPCs. For bidirectional streaming, only message counts (outbound/inbound) are well-supported. Latency metrics are challenging (e.g., 1-hour latency is unhelpful). This is an area for future improvement, and community ideas are welcomed.
20:23 Q&A: Binary Logging for Bidirectional Streaming: Binary logging is supported for bidirectional streaming, logging individual send and receive message events for each message within the stream.