Abstract:

The upcoming Linux 6.18 kernel will incorporate patches aimed at resolving severe system lockups reported when systemd units process large numbers of files on filesystems utilizing the lazytime mount option. The problem arises when a unit exits, forcing the kernel to switch hundreds of thousands or millions of "dirty inodes" (pending timestamp updates) to the parent cgroup's writeback list. The complexity of this sorting operation is currently quadratic in relation to the number of dirty inodes, resulting in 100% CPU utilization that can persist for hours. The fix, included in VFS pull requests submitted by Microsoft Linux engineer Christian Brauner, targets the writeback code to mitigate this performance bottleneck.

Linux 6.18 Will Fix Lockups When Systemd Units Read Lots Of Files

• 06:33 AM EDT (Publication Start): Linux engineer Christian Brauner (Microsoft) submitted 12 pull requests touching the Virtual File System (VFS) portion of the kernel for Linux 6.18, including a fix for system lockups.
• Trigger Condition: Lockups occur when a systemd unit reads a massive quantity of files from a filesystem that is mounted using the lazytime option.
• lazytime Defined: This mount option improves performance by initially updating file access/modify/creation times only on the in-memory inode, delaying the costly on-disk updates until eviction, fsync, or similar operations.
• The Problem: When a systemd unit exits (e.g., a large cron job slice), it can result in "hundreds of thousands or millions" of dirty inodes needing to be moved.
• Technical Breakdown: The internal logic function, inode_do_switch_wbs(), handles sorting these dirty inodes into the appropriate place in the target writeback list (b_dirty). This logic has linear complexity relative to the number of dirty inodes.
• Key Takeaway (Quadratic Complexity): When applied to millions of inodes, the overall time complexity becomes quadratic, causing CPU workers to become pegged at 100% for extended periods (hours) while switching inodes to the parent writeback group (wb).
• Resolution: The patches address this complexity issue in the writeback code and are expected to be fully integrated into the Linux 6.18 kernel merge window.