Unleashing Observability: Enabling Distributed Tracing in gRPC with OpenTelemetry
0:01 Introduction: The speakers, Purnesh Dixit and Sourabh Singh, both work at Google on the gRPC team.
0:36 Why Observability?: Observability helps understand the behavior of distributed systems, debug issues, and identify performance bottlenecks.
1:33 gRPC Observability Intro: gRPC libraries offer built-in observability features, including interceptors that collect raw data (logs, metrics, traces) and export them to respective backends.
3:48 OpenTelemetry (OTel): OTel is a CNCF-incubated, open-source framework that helps create and manage telemetry data, preventing vendor lock-in by providing standard APIs, SDKs, and data formats.
6:30 Understanding Request Lifecycle with Tracing: Tracing helps understand the lifecycle of a request, especially in microservices architectures. It involves sampling requests to avoid excessive overhead in production.
7:56 Tracing Components: Traces track a request's journey, divided into spans representing units of work (e.g., database queries, API requests).
8:51 Visualization: Distributed tracing provides visualizations of request timelines across multiple services.
10:20 gRPC OTel Plugin - Client View: The plugin provides visibility into service communication on the client-side, generating "call spans" (overall gRPC call) and "attempt spans" (individual retry attempts). It captures timing, status, metadata, and propagates the trace ID and span IDs for end-to-end tracing.
12:40 Call Span: Represents the overall gRPC call and contains high-level information from start to end. It's propagated to the next service for distributed tracing.
14:22 Attempt Span: Contains detailed information about a single call attempt, helping pinpoint the root cause of failures. It's not propagated to the next service.
17:24 gRPC OTel Plugin - Server View: The server side has only one "server span" per call, as it doesn't deal with retries. The server receives the tracing context from the client but doesn't send it back.
19:43 Implementation Details: The OTel plugin uses a 29-byte header with a standard key ("traceparent") understood by all OTel vendors. It leverages "TextMapPropagator" to inject/extract span context.
21:30 Client-Side Instrumentation: Developers provide the "TraceTextMapPropagator" when creating the gRPC client to instrument the application.
22:04 OTel Plugin Code: The plugin intercepts the RPC, injects the OTel span context into the header, and handles both "gRPC-trace-bin" (binary) and "traceparent" (W3C context) headers.
24:30 Server-Side Instrumentation: Similar to the client, developers provide the "TextMapPropagator" when creating the server. The server extracts the span context from the incoming request.
26:16 TextMapPropagator: It can be customized, allowing users to bring their own propagator for more control over header handling.
27:10 Simple API: Only two options are needed: "TracerProvider" (OTel SDK) and "TextMapPropagator" (gRPC or custom).
28:11 Plugin Code: The plugin sets the header and span context (client) and decodes the incoming bytes (server).
28:53 Carrier: A simple object with "get" and "set" for handling string key-value pairs in the metadata.
29:54 Current Status: Design is complete, Java has an experimental API (1.68), and C++, Go, and Python are in active development.
30:14 Future Plans: A gRPC Conf is planned for 2025 in Bangalore, India.