This presentation by Mike Cruskll from Google introduces Protobuff Editions, a new framework designed for the controlled, extensible evolution of the Protobuff language and its generated APIs. It addresses the historical challenges of drastic, backward-incompatible changes in previous Protobuff versions (Proto 1, 2, and 3) that led to painful migrations and ecosystem fragmentation. Inspired by Rust Editions, this framework offers fine-grained control over individual language features, ensuring backward compatibility and a clear migration path. Editions allow users to incrementally adopt new capabilities while retaining older behaviors through explicit overrides. The talk emphasizes a new commitment to stability, where released editions maintain consistent behavior, and announces the release of Edition 2024. A successful internal Google migration to Edition 2023 validated the framework's robustness, and the presentation also covers the implications and considerations for open-source ecosystems.
Summarizing Protobuff Editions: A Framework for Controlled Language Evolution
0:16 Challenges with Protobuff Evolution: Protobuff has struggled with evolving its language and APIs despite good schema evolution. Past major versions (Proto 1, 2, 3) introduced drastic, backward-incompatible changes, leading to painful multi-year migrations or ecosystem bifurcation.
2:14 Lessons from Proto 3: Proto 3 introduced significant changes (e.g., implicit field presence, packed repeated fields, maps, JSON support). Reintroducing the optional keyword for explicit presence tracking in 2020 demonstrated that backward-compatible changes are possible but require substantial effort, including compiler/plugin support.
3:36 Introducing Protobuff Editions: To enable more gradual improvements, Protobuff Editions (inspired by Rust Editions) was developed. It's a framework for configuring individual language features at a fine-grain level, designed for controlled and extensible evolution of both the Protobuff language and generated code, rather than creating new monolithic syntax versions.
4:02 Key Benefits & Migration: Editions guarantee backward compatibility and a migration path. Users can upgrade to newer editions for the latest features while retaining old behaviors through explicit feature overrides. This allows for precise, incremental upgrades, ensuring compatibility between files on different editions.
4:49 Edition Declaration & Feature Overrides: Instead of syntax, Protobuff files now declare their edition (e.g., edition = "2023"). Features are specified like ordinary options, but their default values can vary between editions and apply lexically, affecting everything within their scope (file, message, or field level).
6:53 Language Changes in Edition 2023: A notable change in Edition 2023 is the removal of the optional keyword, restoring Proto 2's semantics of tracking field presence by default, after identifying problems with implicit presence.
7:15 API Evolution with Plugin Features: Protoc plugins can define their own language features to evolve APIs. An example is the C++ string type feature, which allows accessors to return string_view instead of string&, enabling future internal memory optimizations without breaking API users.
9:22 Commitment to Stability (Long-Term Support): Once an edition is officially released, its defined behaviors are frozen (outside minor bug fixes). This guarantees consistent generated code and runtime behavior, even with newer Protobuff versions. New features or default changes are introduced only in new editions, giving users control over adoption.
10:28 Proto 2 & Proto 3 as Historical Editions: Proto 2 and Proto 3 are now treated as special historical editions with frozen behaviors and long-term support. They will continue to receive bug fixes and performance improvements but will not gain new features unless upgraded to a newer edition. No forced migrations are planned, but very old editions will eventually reach end-of-life with ample notice and tooling.
12:09 Successful Internal Rollout at Google: Google successfully migrated hundreds of thousands of internal Protobuff files to Edition 2023, validating the framework's robustness in real-world scenarios. A tool called prototiller was developed for programmatic proto transformation, with plans for a future public release.
13:17 Open Source Ecosystem Considerations: In open-source environments, version skew between various components (Protoc compiler, language generators/plugins, runtime implementations, and third-party tooling) poses challenges.
Protoc & Generators: Requires a Protoc version that understands the edition and generators that explicitly advertise their support. Proto owners may delay upgrades for wider compatibility.
Runtimes: Runtimes do not support code generated by newer generators. Upgrading protos in a behavior-preserving way ensures wire-format compatibility with older binaries.
Third-Party Tools: Tools leveraging the plugin framework need to advertise their edition support. Tools that directly parse proto files need updates to understand the new syntax.
18:27 Edition 2024 Released: Edition 2024 is officially available, focusing on incremental improvements for specific user pain points. These include language-specific changes (e.g., Java enum size limitations) and "protoc-end" changes (e.g., enforcing naming style guides during builds) that are easier for third-party tools to implement.
21:04 gRPC and Editions: gRPC, as a Protoc plugin, must advertise its supported editions. However, at present, gRPC does not directly leverage any of the features introduced in Protobuff Editions.
22:51 Compiler as a Library: When asked about a language-specific API for the Protoc compiler, the speaker recommends using protoc to output a file descriptor proto, which can then be dynamically built and consumed as libraries in various languages.