This talk explores the concept of filtering and transforming Git commit graphs to better manage sub-projects, dependencies, and monorepos. The speaker contrasts simple, invertible transformations, such as adding a directory prefix to a project's history, with more complex, non-invertible operations like filtering out specific files or redacting content. The discussion centers on two key questions: whether a transformed commit should retain the original's change ID, and whether most real-world use cases require invertibility for bidirectional synchronization. Through audience interaction, various use cases are identified, including importing/exporting code between repositories, managing projects with different licenses (as in Android), templating changes, and backporting. The talk also touches on significant challenges posed by these transformations, such as breaking commit signatures and potential scalability issues with the Git object database.
Summary of Filtering Commit Graphs
0:00:24 Introduction to Commit Graph Filtering: The speaker introduces the idea of transforming commit histories as a more powerful alternative to git submodule for managing sub-projects and dependencies. This involves rewriting a series of commits according to a defined set of rules.
0:01:09 Simple, Invertible Transformations: The most common use case (estimated at 99%) is adding a directory prefix to all files in a project's history. This allows a repository to be cleanly imported into a subdirectory of another while preserving its history. This operation is easily reversible.
0:03:53 Complex, Non-Invertible Transformations: When a filter removes specific files or commits (e.g., test data or sensitive information), the transformation becomes non-invertible. This complicates workflows that require bidirectional synchronization, as changes cannot be easily sent back to the original source.
0:07:07 Tooling Landscape: The talk references existing tools that perform these operations, including git filter-branch, Google's Copybara, and Josh, which uses a dedicated language to define complex transformations like text replacement and directory mapping.
0:11:30 Key Question 1: Change IDs: A fundamental question is whether a filtered commit should share the same change ID as its original. While this could help trace a change's origin, it creates divergence because the commit objects are different. The general sentiment is that they should have different IDs but include a link to the source change.
0:14:41 Key Question 2: The Need for Invertibility: The speaker questions whether most real-world workflows actually require perfect, invertible transformations. If not, simpler tools focused on one-way imports or exports would suffice for the majority of use cases.
Importing/Exporting Dependencies: The primary use case is moving code between a central monorepo and various external repositories.
License Management (0:17:09): In projects like Android, filtering allows combining code with different licenses from separate repositories into a single cohesive product.
Content Redaction and Rewriting (0:23:01): Filters can be used to remove sensitive information, update include paths, or automatically comment/uncomment code blocks for different build targets (e.g., internal vs. open-source).
Templating and Backporting (0:19:44): Applying templatized sets of changes or automatically adapting a commit to be applied to a different branch (backporting).
0:18:37 Technical Challenge: Broken Signatures: A major issue is that transforming a commit invalidates its cryptographic signature. This breaks the chain of trust unless the original commit can be referenced.
0:27:19 Technical Challenge: Scalability: Merging the histories of multiple repositories into one can create a massive and inefficient object database, which poses challenges for performance and features like partial clones.
0:28:31 Path Forward: The speaker proposes focusing on building a tool that handles the most common 90% of use cases—such as adding prefixes and performing text replacement—without necessarily guaranteeing perfect invertibility.