This presentation introduces S5, an open-source protocol and data structure for content-addressed, decentralized web archiving. The speaker, "Red," highlights the inherent ephemerality of the web and the vulnerabilities of centralized archiving solutions, such as single points of failure and censorship. S5 addresses these issues by promoting a distributed approach that achieves true content deduplication through pre-hashing canonicalization, leveraging Merkle trees and the Blake3 hash function for efficient, verifiable data streaming and parallel processing. A key innovation is the separation of archive metadata from the actual content, enabling the public sharing of verifiable metadata while allowing individuals to privately control their content blobs, thereby navigating legal complexities and fostering a globally collaborative digital memory. The talk includes a live demonstration of local web archiving and browsing, discusses planned future enhancements like historical tracking and compression, and outlines a layered trust model for distributed access to archived data.
Summarizing the S5 Decentralized Web Archiving Project:
0:00:14 Introduction to S5: "Red," an open-source developer, introduces the S5 project, a protocol and data structure focused on content-addressed application fields, with a primary goal of decentralized web archiving.
0:00:36 The Problem of Web Ephemerality: Critical information, scientific data, and cultural heritage are frequently lost from the web due to company closures, hosting cost issues, or deliberate deletions (e.g., government data removal).
0:01:10 Limitations of Centralized Archiving: While institutions like the Internet Archive (Archive.org) are vital, their centralized nature creates single points of failure, vulnerability to legal pressures from specific countries, funding challenges, and potential for censorship or content removal requests (e.g., for privacy reasons).
0:02:49 The "Do-It-Yourself" Archiving Approach: If reliance on a single organization is insufficient, the solution is for individuals to take archiving into their own hands, utilizing a range of existing open-source tools (e.g., PWB, Börsler, Wagbooks, Heretrix, YouTube DLP, Archive Box).
0:04:31 WARC Files and Their Inefficiencies: Web ARChive (WARC) files capture complete snapshots of web requests but suffer from a flat, often redundant data structure, lacking efficient deduplication. Minor variations in HTTP headers or timestamps prevent identical content from being recognized as such.
0:06:50 Content Addressing for True Deduplication: S5 implements content addressing, inspired by systems like Git and BitTorrent, to achieve real deduplication. This involves decompressing (e.g., gzip) and canonicalizing HTTP responses before hashing, ensuring identical content is stored only once, regardless of how it was initially served.
0:08:21 Resilience, Verifiability, and Parallel Hashing with Blake3: The system offers resilience and verifiability through cryptographic hashes, allowing untrusted sources to provide data that can be verified against a trusted root hash. S5 uses the Blake3 hash function, which is fast, secure, and uniquely parallelizable due to its Merkle tree structure, enabling efficient, verifiable streaming of large files.
0:12:06 S5 Protocol Components: The S5 protocol consists of three core components: a Blobstore (an abstraction for deduplicated content storage with various backends like local filesystems or S3), a CBOR-encoded file system structure (mapping content hashes and metadata to paths), and an update mechanism using ED25519 public keys to sign new versions of immutable file systems.
0:14:14 Rich Metadata for Archived Content: The generic file system structure not only stores content hashes but also comprehensive web archive metadata, including IP address, HTTP version, request/response headers, and timestamps, allowing for detailed historical context.
0:15:34 Live Demonstration: A demo showcases the process: scraping mmcd.net using the Internet Archive's Seno tool, importing the generated WARC file into the local S5 Blobstore with S5 Import, and then browsing the archived website locally via S5 Surf Webarchive, demonstrating successful content retrieval.
0:18:52 Addressing Legal Challenges through Metadata Separation: S5 proposes a legal solution for decentralized archiving by decoupling content blobs from metadata. Individuals can legally archive content locally and publish only the metadata (hashes, directory structure), fostering a "trustnet" where multiple parties can attest to the validity of content without directly hosting potentially copyrighted material.
0:21:25 Planned Future Enhancements: Key areas for improvement include implementing full history tracking, enhancing storage efficiency for small files (e.g., via key-value stores), and adding a transparent compression layer within the Blobstore.
0:22:45 Vision: A Global Collaborative Memory: The long-term vision is a layered, decentralized system where individuals archive locally and share with progressively wider networks (friends, trusted organizations, public DHT trackers), creating a unified, mergeable, and comprehensive global web archive.
0:25:10 Expanded Use Cases and Indexing: S5 can integrate with other large datasets like Common Crawl (by indexing pointers without hosting the massive data), incorporate media archives from YouTube DLP, and mirror HTTP resources (e.g., Linux distribution mirrors, CDNs) to be streamed from within the archive.
0:27:08 Current Implementation Status: The local archiving and browsing functionality demonstrated in the video is fully implemented. Collaborative features like peer-to-peer connectivity and the blob hosting protocol are partially developed but require further integration for public use.