This expert panel discussion from Mastercard delves into the adoption and implementation of gRPC bidirectional streaming for their critical financial systems, which process billions of transactions globally. The panel, comprising leading engineers, explores the benefits of gRPC, such as enhanced security, performance, developer productivity, and edge integration, citing its alignment with Mastercard's core technological principles. They also transparently discuss significant challenges encountered, including achieving robust observability across multiple protocol layers, adapting to client-side load balancing for persistent connections, and rigorous security validation for stateful communication over HTTP/2 and TLS. The session concludes by outlining future strategies, including the adoption of XDS for service discovery and an exploration of HTTP/3 QUIC for a lighter, faster, and more resilient protocol.
Evolving Critical Systems with gRPC Bidirectional Streaming at Mastercard
0:00 Introduction & Mastercard's Scale: Prasana Shinuasan, Director of Software Engineering, introduces the panel and highlights Mastercard's vast global financial network, processing 159.4 billion transactions in 2024. The discussion focuses on gRPC's role in their event-driven architecture.
0:15 Panelists: The panel includes Ravi Cherukuru (Payments Architect), Thiagar Rajin (Developer Productivity Architect), Jenny Zang (Distinguished Engineer), and Eric Robbins (Security Architect).
2:27 Ravi (Payments Architect): Mastercard's custom-built, low-latency tech stack evolved towards gRPC to build a more scalable payment network, leveraging its security, developer experience, and faster delivery for hundreds of millions of daily transactions.
4:22 Thiagar (Developer Productivity Architect): gRPC was foundational for modernizing critical systems due to its alignment with Mastercard's principles: security, performance, developer productivity (protobuf code generation, clear contracts, multi-language support), and crucial edge integration over HTTP/2 for fine-grained access control.
5:50 Fintech Factor & Performance: The seamless, asynchronous flow of continuous transactions (e.g., tapping a phone at a merchant) makes bidirectional streaming ideal for high throughput, low latency, and event-driven architectures with minimal overhead, offering superior scalability over REST or WebSockets. Performance is a necessity in fintech, where every millisecond counts.
7:46 Jenny (Distinguished Engineer): Emphasizes the importance of observability, which was a significant learning curve. Mastercard invested heavily in telemetry at every layer (TCP/IP, HTTP/2 streaming, application) to monitor traffic and connection health (especially persistent bidirectional connections) between client and server for proactive issue identification and traffic rerouting.
10:23 Scalability Challenges with gRPC: Traditional instance-scaling methods don't work well due to gRPC's persistent connections. This necessitates new strategies for horizontal scalability and traffic management across regions.
11:29 Eric (Security Architect): Highlights the need for gRPC to align with Mastercard's strict security standards, driven by regulations and customer mandates. This required deep analysis of the protocol down to layers 3 and 4 to identify and address any security gaps.
13:16 Eric (Security Architect): The biggest shift was from stateless REST APIs (HTTP/1.1) to gRPC's connection-based state. Unlike stateless REST where every request is re-authenticated/authorized, gRPC bidirectional streaming might authenticate only once per channel.
15:57 Security Validation: Mastercard meticulously validated gRPC's security by using a TCP proxy to modify individual bytes of TLS records during a gRPC bidirectional stream. This proved that any interruption would result in a fatal TLS alert, closing the connection and forcing re-authentication upon reconnection, thus maintaining cryptographic binding and preventing tampering. They propose showing this validation process in a future session.
18:46 Jenny (Distinguished Engineer): Discusses load balancing challenges. Traditional server-side load balancers (like F5, routing at TCP layer) don't work for gRPC's persistent connections, leading to overloaded instances while others remain idle.
20:29 Client-Side Load Balancing: Mastercard shifted to client-side load balancing, embedding intelligence within the client to manage connections, distribute traffic evenly across available streams and channels, and dynamically adapt to backend server health. This is a "per message" based load balancing.
22:17 Ravi (Payments Architect): Addresses customer viewpoint concerns. gRPC reflectors, while useful internally for API onboarding, can expose internal methods, requiring careful security for external APIs. Challenges in service/endpoint discovery and load balancing still exist.
24:00 Thiagar (Developer Productivity Architect): Future plans include coupling XDS (External Discovery Service) for dynamic configuration, service discovery, and failover with a patented adaptive request handling mechanism that monitors channel and stream health.
24:41 Exploring HTTP/3 QUIC: Mastercard is investigating HTTP/3 QUIC, built on UDP, for a lighter, faster, and more resilient protocol. Its decoupled streams avoid TCP's head-of-line blocking, and its 0-RTT security is expected to significantly improve speed and resilience to connection failures.
25:40 API Catalog: Mastercard uses tens of internal APIs, not yet externally exposed by design. They use customizable internal style guides rather than Google's AIPs.
26:24 Developer Productivity & Front-End Integration: Mastercard uses Protobufs as a standard with a central schema repository for strong-typed APIs and clear contracts, reducing ambiguity early on. Protocol remediation layers exist for lower TCP/gRPC layers. They clarify that their current gRPC usage is for service-to-service communication with trusted clients, not direct UI integration.
27:00 Load Balancing (Message vs. Channel): Client-side load balancing is done on a "per message" basis. The client's logic (library) identifies active streams/channels across regions and balances traffic intelligently, ensuring no server affinity due to redundancy and unique message IDs for traceability.
31:01 Client-Side Load Balancer Placement: The client-side load balancing logic is implemented in a library that can sit anywhere (internal or external clients) within Mastercard's existing network infrastructure. The goal is a unified interface for all customers in the future.