Abstract:

This video analyzes the catastrophic data loss incident at Kyoto University on December 14, 2021, where an HP Enterprise (HPE) managed supercomputer lost 77 terabytes of research data. The root cause was a subtle but critical flaw in the deployment process of a Bash housekeeping script designed to delete old log files.

The incident is broken down as a technical case study, explaining the difference between interpreted languages like Bash and compiled languages. The core of the problem was a race condition created when the running cleanup script was updated using an "in-place overwrite" (cp command) rather than an "atomic swap" (mv command). The script had already loaded an old variable name for the log directory path into memory before the update occurred. When it resumed execution, it read the new version of the script, which referenced a new, renamed variable. Because this new variable was unset in the script's memory, Bash interpreted it as an empty string, causing the delete command to target the supercomputer's root file system instead of the specific log directory. The analysis concludes by detailing the impact of the data loss and outlining preventative measures, such as proper deployment methods and writing safer Bash scripts.

Anatomy of a Disaster: The Kyoto University Data Deletion

• 0:00:22 The Incident: On December 14, 2021, researchers at Kyoto University discovered that months of data had vanished from their supercomputer. A faulty script had deleted 77 terabytes and 34 million files, affecting 14 different research groups.
• 0:00:42 The System: The data was stored on a Luster parallel file system, a high-performance solution for supercomputers. The system's hardware and certain management tasks, including backups and cleanup, were handled by HP Enterprise (HPE).
• 0:01:56 The Culprit: The deletion was caused by a Bash script provided by HPE, which was intended to perform routine housekeeping by deleting log files older than 10 days.
• 0:03:42 How Bash Reads Scripts: Unlike compiled languages, Bash is an interpreted language that reads and executes a script file incrementally. It uses a file pointer to track its position, meaning if the file's content changes mid-execution, Bash will continue reading from its current position in the newly modified file.
• 0:04:32 The Critical Flaw (In-Place Overwrite): The script was updated using the cp (copy) command. This performs an "in-place overwrite," which modifies the contents of the existing file (inode) directly. Any process that has the file open will immediately see the new content.
• 0:05:27 Atomic Swaps vs. In-Place Overwrites: In contrast, the mv (move) command performs an "atomic swap." It points the filename to a completely new file (inode), leaving the original file intact for any processes that are still using it. This would have prevented the incident.
• 0:07:21 The Race Condition Explained: The script was running and had already processed the line defining the log directory path (e.g., LOG_DIR_V1="/path/to/logs"). While the script was busy with other tasks, HPE deployed the new version, which used a renamed variable (e.g., LOG_DIR_V2="/path/to/logs").
• 0:08:09 Deleting Everything: When the script resumed, it executed the deletion command from the new file (rm -rf $LOG_DIR_V2). However, since LOG_DIR_V2 had never been defined in the currently running process, Bash interpreted it as an empty string. The command effectively became rm -rf /, targeting the root of the file system and deleting everything it could.
• 0:08:55 The Aftermath: The deletion ran for nearly two days before being stopped. Although backups existed, only 49 TB of the 77 TB deleted could be recovered. 28 TB of data were permanently lost, forcing four research departments to lose all their work.
• 0:10:13 Key Takeaways & Prevention: HPE took full responsibility. The incident could have been prevented by using mv instead of cp for script updates or by using the set -u flag in Bash, which causes the script to exit with an error if it encounters an undefined variable.