This webinar provides a comprehensive guide to debugging race conditions and other concurrency issues in multi-threaded C++ applications. The session is divided into two parts. First, David, a software engineer from AWS, presents a detailed case study of a complex, intermittent race condition within the AWS Database Migration Service. He explains how a bug involving mismatched component lifecycles and memory pools led to a use-after-free error, and how time-travel debugging was instrumental in diagnosing the root cause.
In the second part, C++ debugging expert Greg Law from Undo offers a broader survey of tools and techniques. He defines race conditions, explains why they are difficult to solve, and provides hands-on demonstrations of various tools including Valgrind (Helgrind), Thread Sanitizer, and Undo's time-travel debugger (UDB). The presentation also covers the nuances of the C++ memory model, the limitations of static analysis and mutexes, the potential of AI-assisted debugging, and advanced GDB features for handling multi-threaded programs.
Tools and Techniques for Debugging Multi-threaded C++
0:00:50 A Real-World Race Condition: David from AWS presents a case study from the Database Migration Service (DMS) where an intermittent bug caused connection failures with a "gibberish" address, indicating memory corruption.
0:03:35 Identifying the Corrupted Data: The issue was traced to a shared data structure containing the database address, which was being corrupted by another thread.
0:04:41 Capturing the Bug with Time-Travel Debugging: To debug the non-deterministic issue, the team used Undo to record a failing execution. This allowed them to repeatedly analyze the exact moment of failure using memory watchpoints and reverse-debugging commands.
0:08:38 The Root Cause: Mismatched Lifecycles: The bug was a use-after-free error. A short-lived thread fetched the database address and stored it in its own memory pool. When this thread terminated, its memory pool was freed, leaving a dangling pointer in the shared data structure that other threads continued to use.
0:12:10 Key Takeaway: A component with a long lifecycle (like a shared credential manager) should not depend on a resource owned by a component with a short lifecycle (like a temporary worker thread).
0:12:31 The Solution: The fix was to decouple the lifecycles by giving the long-lived credential data its own dedicated memory pool that would persist for as long as it was needed.
0:17:42 Defining Race Conditions: Greg Law explains that race conditions are not limited to threads accessing shared data. They can involve any concurrent operations (e.g., multiple processes, user input, OS interactions) on a shared resource (e.g., the file system) where the order of operations affects the outcome.
0:21:03 Tool Demo - Valgrind (Helgrind): Valgrind's Helgrind tool is demonstrated to dynamically detect data races in a sample C++ program without needing to recompile the code.
0:23:08 Tool Demo - Thread Sanitizer (TSan): As a higher-performance alternative, Thread Sanitizer is shown to find the same data race. It works as a compiler feature (-fsanitize=thread) and instruments the code during compilation.
0:28:43 A Mutex is Not Always the Fix: Adding a mutex can resolve a data race (making tools like TSan and Valgrind report no errors) but may not fix the logical race condition. The program's output can still be non-deterministic and incorrect, depending on which thread acquires the lock first.
0:37:14 Tool Demo - Time-Travel Debugging: Using Undo's Live Recorder and UDB, a single intermittent failure is captured. The recording can then be debugged deterministically, allowing the developer to step forwards and backwards in time to precisely observe the sequence of events that led to the bug.
0:42:19 AI-Assisted Debugging: AI tools can analyze code statically, but their real power comes from integration with dynamic analysis. The demonstration shows an AI explaining the root cause of the failure by analyzing the captured execution trace from the time-travel debugger.
0:45:26 Advanced GDB for Threading: GDB is a powerful tool for multi-threaded debugging. The scheduler-locking on command is explained, which freezes all other threads while stepping through the code, preventing the state from changing unexpectedly.
0:56:40 Diagnosing Corrupted Stacks: If a core dump produces a garbled or missing call stack, it's a sign of stack corruption. Time-travel debuggers are highly effective for these bugs because they reconstruct the execution history without relying on the integrity of the corrupted stack.