This presentation explores the extensive application of Git, augmented by the Git-Annex extension, for comprehensive personal data management beyond traditional code versioning. The speaker, Yann Büchau, details how this combination can fulfill an "ideal world" scenario for managing documents, photos, notes, and other personal files, offering benefits like robust change history, decentralized backups, offline accessibility, and user control over data, while rejecting proprietary cloud solutions. The talk contrasts Git-Annex with simpler synchronization tools like Syncthing and even Git LFS, highlighting its superior capabilities in handling large files, managing diverse storage endpoints, ensuring data integrity through checksums, and offering fine-grained control over file distribution. Practical demonstrations include integrating Git-Annex with a self-hosted Git forge (Forgeo Anexo) and various text-based tools for tasks such as note-taking, time tracking, password management, and financial accounting, showcasing a powerful, open-source ecosystem for safeguarding and organizing one's entire digital life.
Exploring "Life in Git" with Git-Annex: A Comprehensive Data Management System
0:57 Introduction to the Speaker and Topic: Yann Büchau, a Linux user and researcher in atmospheric physics, introduces the concept of using Git for managing one's entire digital life.
2:09 The Problem with Traditional File Management: Files (documents, images, notes, measurement data) are often disorganized across various apps and cloud services, leading to issues with versioning, backups, and data control.
2:42 The "Ideal World" for File Management: An ideal system would provide complete change history (including proof of origin), allow metadata tagging, support collaboration, enable offline work, ensure sufficient backups (following rules like 3-2-1), and grant full user control over data without relying on proprietary services.
5:21 Rejection of Proprietary Solutions: Services like Dropbox, Google Drive, and OneDrive are deemed unsuitable due to their lack of user control over personal data.
5:40 Limitations of Simple Sync Tools (e.g., Syncthing): While easy to install and useful for synchronizing folders across devices, Syncthing is explicitly stated not to be a backup solution as it lacks versioning and propagates deletions. It also doesn't offer native encryption.
6:51 Git for Files – Benefits and Challenges: Git provides strong collaboration and backup capabilities for files by tracking all changes, and encryption is possible. However, its main limitations for general file management are poor handling of large files, a lack of fine-grained control over what gets cloned, and a non-beginner-friendly interface.
8:22 Introducing Git-Annex as the Solution: Git-Annex is a plugin that teaches Git to manage large and non-text files efficiently, and much more.
8:41 How Git-Annex Works: It moves large files out of the Git repository into a dedicated Git annex Objects folder, with Git tracking only symbolic links. Git-Annex maintains metadata, checksums, and knows which computer holds which file, preventing accidental data loss.
9:40 Diverse Endpoints: Git-Annex supports a wide range of storage locations, including SSH, S3, Android devices, and IPFS, with encryption options available.
10:02 Simplified Operations: The git annex assist command automates Git operations (commit, pull, merge, push), making it easier for users less familiar with Git.
10:32 Remaining Limitations of Git-Annex: It still struggles with simultaneous collaboration on the exact same large file (leading to duplicates), remains somewhat complex for beginners, can be slow with millions of files, and has compatibility issues with certain filesystems (like FAT32 or NTFS). It also doesn't inherently care about file timestamps.
12:16 Forgeo & Forgeo Anexo – A Self-Hosted Git Forge: Forgeo is an open-source, self-hostable Git forge (like a personal GitHub) that, with the "Forgeo Anexo" extension, can directly display and manage large files stored with Git-Annex, including visual diffs for images.
13:40 Document Management: Synchronizing family document folders.
13:59 Measurement Data Management: Used for scientific data from drone campaigns.
14:05 Media Database: Managing photo collections, tracking file locations across various storage devices (git annex info), retrieving specific files (git annex get), moving files (git annex move) while enforcing backup policies, and verifying data integrity (git annex fsck).
15:16 Fine-Grained Control: Allows users to specify which files to retrieve based on properties like size or filename patterns.
15:44 NB (Notes in Terminal): A text-based note-taking system that syncs with Git.
15:57 Alexin-Log: The speaker's custom time-tracking tool, using Git-Annex as a conflict-free backend.
16:36 Pass (Password Manager): A standard Unix password manager that uses GPG-encrypted text files, easily synchronized via Git, and supporting OTP codes.
17:15 Plantext Accounting: Tools for tracking finances in plain text files (e.g., Ledger, Beancount), easily imported from bank CSVs and versioned with Git for tax purposes.
18:01 Calendar Sync: CalDAV servers can be configured to track calendar data changes with Git, providing backups and easy recovery.
18:29 Broad Applicability: Any data that can be represented as plain text or deterministically generated from text (e.g., database dumps) can be effectively managed with Git, and Git-Annex for larger binary files.
The speaker's NixOS configuration is publicly available on his GitHub.
Git-Annex allows deletion of old versions from the annex, although file hashes remain.
Git-Annex is considered "orders of magnitude better" than Git LFS for large file management due to its advanced features and fine-grained control.
There are no hard size limits for files (e.g., 20 petabyte files are theoretically possible), though very large files take longer to hash. Performance can slow with millions of files, but Git-Annex provides control to manage this.
Git-Annex is excellent for syncing data across multiple hard drives, using hashes to optimize the process and avoid re-copying existing files.