This presentation details Anthropic's journey to escape the scaling limitations of standard Docker and OCI image formats, necessitated by the rapid development cycle of large, frequently compiled software for AI research. The initial deployment relied on a single 35 GB Docker image, resulting in painfully slow 4-minute pull times and hourly rebuild cycles.
The solution involved shifting away from OCI images entirely, utilizing Nix derivations as the distribution format and leveraging a robust, replicated internal Nix store. This infrastructure reduced startup pull times for large workloads from an estimated 70 seconds to 15 seconds. To further eliminate wasted bandwidth and achieve the near-zero-second pull times seen with local Docker caches, the team developed Nix Sidecar. This solution, implemented as a Kubernetes init/sidecar container, manages host SSD caching, fetches only necessary Nix store paths, and uses bind mounts to provide the application with a highly specific and efficient read-only environment, all without requiring customization of the underlying ContainerD runtime or managing a persistent daemon service on the host.
Escaping Docker: How Anthropic Uses Nix Sidecar for Scalable Builds
0:18 Anthropic Background: The speaker, Anish, works on developer experience at Anthropic (AI Safety Research, makers of Claude).
0:58 The Docker Problem: Anthropic used a single Docker image that grew to 35 GB (uncompressed) due to ML libraries and accumulation of research dependencies.
1:51 Pain Points: The large image caused 4-minute pull times and nearly 1-hour rebuilds, preventing continuous integration (CI) and requiring manual build coordination.
2:28 Docker Scaling Issues: Dockerfiles lack sufficient abstraction, parallelism is unwieldy (multi-stage), and linear layering means changing one root dependency (e.g., Cuda) forces massive rebuilds.
5:03 OCI Layer Limit: The OCI standard (used by Docker) limits images to 128 layers, preventing the fine-grained dependency management needed for large applications.
5:41 ContainerD Bottleneck: ContainerD, the runtime, extracts image layers serially after parallel downloading, creating a performance bottleneck for high layer counts.
6:03 The Nix Store Solution: Anthropic escaped the Docker/OCI paradigm for a new compiled software initiative, building with Nix and uploading products directly to an internal Nix store. Kubernetes jobs fetch the necessary Nix store paths at startup.
9:20 Performance Improvement (Initial): By using a geographically replicated Nix store and caching layers, the required application data pull time for an 11 GB workload dropped from an estimated 70 seconds (Docker) to 15 seconds.
9:52 The Caching Gap: Unlike Docker, which achieves 0-second pulls when an image is cached on the node, the Nix implementation re-fetched data every time, wasting significant bandwidth.
11:46 Rejected Solution: Nix Layered Store (RFC 152): Rejected because it lacked specificity (exposing every path cached on the host) and lacked garbage collection support for the lower (host) store.
13:48 Rejected Solution: Nix Snapshot: Rejected despite being a strong solution (it supports specificity and GC) because Anthropic could not customize the core ContainerD runtime on all cloud provider AMIs.
16:30 The Solution: Nix Sidecar: A custom, user-space implementation running as a Kubernetes init/sidecar container. It mounts the host SSD, downloads required paths, and uses bind mounts to present only the necessary read-only paths to the main application container (ensuring specificity).
19:03 Mount Cleanup: The Sidecar pattern is essential to ensure bind mounts are cleaned up after the main container exits, avoiding Kubernetes issues.
19:31 Key Takeaway: Nix Sidecar successfully provides node-level caching, achieving 0-second pull times for subsequent jobs on the same node, greatly reducing bandwidth waste and scaling costs.