This talk addresses the increasing need for robust metadata versioning in modern software development, arguing that current mechanisms are inadequate for emerging use cases. The speaker, a software engineering manager at Google, outlines the limitations of existing metadata storage methods like commit descriptions, separate files, and Git notes. These methods fail to support complex requirements for new metadata types, such as supply chain security attestations (e.g., SALSA), detailed code generation provenance, fine-grained licensing information, and handling for large binary assets.
The presentation identifies the core challenge: different metadata types require distinct properties regarding granularity (per-commit vs. per-byte), mutability (immutable vs. append-only), propagation (how metadata evolves with code changes), and exchange (local vs. shared). The speaker proposes that the Jujutsu (JJ) version control system, due to its early adoption stage, workflow-centric design, and extensibility, is the ideal platform for the community to experiment with and develop a flexible, powerful new metadata framework. The talk concludes not with a prescribed solution, but as a call to action for the community to collaboratively design these future systems.
Versioning Metadata: Challenges and a Path Forward with JJ
0:01:45 Current Metadata Limitations: Today's version control systems handle metadata through insufficient means like commit object headers, separate files (e.g., CODEOWNERS), and embedding data in commit descriptions, which is a common but clunky workaround.
0:04:34 Emerging Metadata Needs: There is a growing demand to version control more complex information that is not well-supported by current tools. Key use cases include:
Code Provenance: Detailed information on how code was generated, including the tools, versions, and inputs used.
Licensing: Tracking license information on a per-file or even sub-file level.
Binary Assets: Storing information for external diff and merge tools to handle large binary files, a common problem in game development.
0:09:38 The Core Problem: Diverse Metadata Properties: A one-size-fits-all solution is not feasible because different metadata types require different behaviors regarding:
Granularity: Whether the data applies to a commit, a file, or down to a single byte.
Mutability: If the data is fixed (immutable) or can be changed or appended to (e.g., review comments).
Propagation: Whether metadata should be carried forward and merged across subsequent versions of a file.
Exchange: If the metadata is local-only, always pushed and pulled with the repository, or shared conditionally.
0:13:43 Why Existing Solutions Are Inadequate: Current mechanisms lack flexible control over these properties, impose size limits, are not scalable for large repositories, can be fragile (e.g., destroyed by a rebase), and don't allow for features like lazy fetching of metadata.
0:17:20 Proposing JJ as a Testbed: The speaker suggests that Jujutsu (JJ) is the ideal platform for innovating on metadata solutions due to its early adoption stage, workflow-first design philosophy, and inherent extensibility through APIs and custom backends.
0:20:16 A Call to Action for the Community: The talk intentionally does not offer a specific implementation. Instead, it serves as a call for the open-source community to discuss and collaboratively design a new, flexible metadata system that works for everyone, rather than a solution tailored only for Google.
0:22:37 Priority Use Case: In the Q&A, the speaker highlights that tracking the provenance of auto-generated code is a primary and pressing use case. Google currently addresses this with a complex, custom internal system that is not open-sourceable.