Lucas, a protocols engineer at Cloudflare, discusses the journey and implementation of QUIC and HTTP/3 to enhance gRPC at Cloudflare's scale. The talk details the standardization of QUIC (RFC 9000) as a transport protocol independent of HTTP, emphasizing its integrated security, stream independence, and improvements over TCP/TLS. He highlights Cloudflare's internal development with the Rust-based 'kiche' library. The presentation contrasts the historical challenges and security vulnerabilities of gRPC over HTTP/2 with the inherent resilience and performance benefits of HTTP/3/QUIC, particularly its immunity to certain denial-of-service attacks and head-of-line blocking. Preliminary performance data from lab tests and customer deployments show significant latency reductions and improved success rates with HTTP/3. Finally, Lucas touches on the future potential of running QUIC stream abstractions over various bidirectional transports for diverse use cases.
Summarizing "Bringing HTTP/3 to gRPC at Cloudflare Scale"
0:00 Introduction & Speaker's Background: Lucas, a Cloudflare protocols engineer and co-chair of the QUIC Working Group, introduces his talk on bringing HTTP/3 to gRPC at Cloudflare's scale, drawing on a decade of experience in protocol development.
0:16 Cloudflare's Role: Cloudflare aims to build secure and performant connectivity between clients, origins, and data centers globally, acting as a CDN (Content Delivery Network) that processes various traffic types, including gRPC and websockets.
0:41 Talk Agenda: The agenda includes background on QUIC and HTTP/3, their implementation at Cloudflare, the introduction of gRPC over HTTP/2, challenges with HTTP/2, opportunities with HTTP/3, and a wrap-up.
0:27 QUIC Fundamentals: QUIC is a standardized (RFC 9000) transport protocol, distinct from TCP and TLS, integrating security based on the TLS model. It's designed to run over UDP, offers faster handshakes (saving 1-2 RTTs), and provides multiplexed, secure transport that is not exclusively tied to HTTP.
0:59 HTTP/3 Fundamentals: HTTP/3 is built on QUIC, offering similar capabilities to HTTP/2 (streams, frames, header compression) but leveraging QUIC's transport features. It is feature-compatible with HTTP/2 and standard HTTP.
0:40 Core Specifications & Benefits: Key QUIC RFCs include 9000 (transport), 9001 (TLS), and 9002 (loss detection/congestion control). Key benefits include encrypted packets (preventing "oification" by middleboxes), truly independent streams (avoiding head-of-line blocking in lossy networks), and connection migration (useful for mobile workloads).
0:57 QUIC/HTTP/3 Standardization Timeline: Google began working on QUIC before 2015, with the IETF working group forming in 2016. RFC 9000 was published in 2021, and HTTP/3 (RFC 9114, 9115) in 2022, after extensive real-world testing.
1:41 Cloudflare's QUIC/HTTP/3 Implementation: Cloudflare initiated beta support in 2018, open-sourced its Rust-written QUIC/HTTP/3 library "kiche" in 2019, and made it available to early adopters the same year. In 2024, they open-sourced "H3I" for testing and debugging.
1:50 Cloudflare's Proxy Chain: Cloudflare operates a chain of proxies (ingress, brain, caching, egress). "kiche" powers the QUIC implementation at the ingress for TLS termination.
2:12 gRPC over HTTP/2: Cloudflare launched gRPC over HTTP/2 support in 2020. This initially required a "crazy scheme" using gRPC web conversion between ingress and egress due to legacy HTTP/1.1 internal proxies.
2:16 Challenges with HTTP/2: HTTP/2 has known vulnerabilities, including "Ping Flood" (2019) and "Rapid Reset" (2023), which are denial-of-service attacks that exploit legal but unexpected uses of the protocol. HTTP/2 is considered "broken in many ways."
2:31 QUIC's Attack Immunity: QUIC is immune to many HTTP/2 attack styles by design, offering a more inherently secure platform for gRPC services. Debugging QUIC is harder due to encryption but tools like Qlog and H3I are being developed by the community.
2:41 gRPC over HTTP/3 Support: An issue for gRPC over HTTP/3 was opened in 2019, with a design proposed and merged in 2021. Tools like Buff can now use an HTTP/3 flag to work with Cloudflare without requiring origin changes.
2:55 Performance Benefits: Lab testing showed gRPC over HTTP/3 could save an average of one Round-Trip Time (RTT) (5-10ms) and up to five RTTs at the P90 percentile. Customer data from a public-to-private (CDN edge) deployment indicates better P75, P95, and success rates for HTTP/3 compared to HTTP/2.
2:49 Future of QUIC: Lucas highlights ongoing work in the IETF to enable the QUIC stream abstraction to run over any bidirectional transport (e.g., RDMA, Unix sockets), expanding its use beyond UDP to potentially improve intra-data center gRPC workloads.
3:01 Q&A - MASQUE Protocol: MASQUE (Multiplexed Application Substrate over QUIC and HTTP/3) is an IETF working group focusing on proxying various network protocols (UDP, IP) over HTTP, allowing for secure tunnels for gRPC over these proxied connections.
3:08 Q&A - Connection Failure Detection: QUIC's loss detection is similar to TCP's (retransmission timeouts) but offers more explicit error codes and a built-in idle timeout for silent connection closure. Future extensions might allow for more rapid failure detection.
3:17 Q&A - Performance in High Retransmission Networks: No personal tests have been conducted on high retransmission rate performance. It largely depends on specific algorithms and implementations, which are more diverse in QUIC than in standardized TCP. The benefit of independent streams in QUIC is that a loss on one stream does not block all others, potentially improving perceived latency in highly concurrent applications.