Greg, Co-founder and Chief Exec at Undo, on Race Conditions

• 00:00:20 Intro: Greg discusses race conditions and debugging. He noticed that marketing content about race conditions gets more traction.
• 00:01:19 Problem: There's a significant problem with underserved needs in terms of race condition debugging.
• 00:02:08 Definition: A bug is when a program's reality diverges from expectations.
• 00:02:47 Factors Making Bugs Hard to Find: The time between the bug and noticing it, and how deterministic the failure is. Race conditions are non-deterministic and have a long time between the problem and noticing it.
• 00:04:19 Race Condition Definition: Concurrent threads of execution acting on a shared resource, where the ordering of operations affects the output. It could involve threads, processes, or even a single-threaded process racing with the outside world (example: file existence check).
• 00:07:01 Tools for Debugging: Thread sanitizer, Valgrind (Hellgrind), GDB, lightweight logging, and time travel/thread fuzzing (Undo).
• 00:07:37 Example Code: Simple C++ code with two threads assigning values to a shared variable, illustrating a race condition.
• 00:08:47 Thread Sanitizer: Using -fsanitize=thread during compilation can detect data races, but requires setting vm.map_random_bits=30 on some systems to avoid errors. Reports conflicting writes and thread backtraces.
• 00:12:15 Thread Sanitizer and Address Sanitizer: Incompatible and require separate builds. Thread Sanitizer checks for proper locking during concurrent data access.
• 00:12:27 Lock Ordering: Detects potential deadlocks from inconsistent lock order.
• 00:13:46 syn sleep by synchronization: Check if thread sleeps during synchronization.
• 00:13:48 Valgrind (Hellgrind): A suite of tools that use binary instrumentation at runtime. No recompilation is needed.
• 00:16:41 Demonstration: Uses a program with four worker threads updating shared characters to show performance differences between instrumentation methods.
• 00:17:16 Race Condition in C: Argument whether C has a race condition or not.
• 00:17:37 Thread Sanitizer Demo: Performance different is observable in both the Valgrind tool suite and Thread Sanitizer.
• 00:19:25 Mutexes and Race Conditions: Adding mutexes might make thread sanitizer say that the code is fixed, while the code still has a race.
• 00:21:15 Undefined Behavior: Data races in C++ result in undefined behavior, meaning anything can happen.
• 00:21:35 Code Still Being Racy: Even if it's welled defined in the source of truth.
• 00:21:37 Canonical Race Condition: Incrementing some shared data, depending on the order.
• 00:24:11 Data Race vs Race Condition: Data races is a good thing to start with for debugging.
• 00:25:48 Testing Assumption: Run it through something like a data race and see if there is something else wrong there.
• 00:26:43 Debugging Tip: Run stress or something in the background to see if the non-deterministic bug is more likely to happen.
• 00:26:51 GDB: GDB can cause the race condition to go away, compiling debug information doesn't change anything.
• 00:29:02 Pthread Set: Small tool called set that's given it's worker threads a name so it can load to debug.
• 00:29:52 Problem: People don't actually know how to drive the application to make it work.
• 00:30:01 Tool: Pthread is a function to give a worker thread a name so it's loaded inside GDB.
• 00:30:29 Thread 2: Thread function where the threads can't run all the threads so it deadlocks.
• 00:31:39 Solution: Print takes up a Mutex.
• 00:32:39 GDB - Set Scheduler Locking: The solution for when "next or step over" something happens.
• 00:34:39 Lip C lock weight: GDB can't be found, so it's one of the internal Lipsy lock that print is going to take.
• 00:34:59 Recommendation: Set thread scheduler locking is useful.
• 00:37:23 Good Starting Point: Sanitize to thread run it and if it finds it, you're done.
• 00:37:40 Assertions: If you run sanitize or helr it can pinpoint where the problem is.
• 00:37:40 Other Scenario: If they don't, switch to GDB.
• 00:39:29 Set Schedu Locking: Only applies for the selected thread.
• 00:40:37 Command Thread Apply All: Will help do command or all or 2 -4 on certain thread for printing stack pointer.
• 00:41:04 Colonel is Dealing: Once you have all of that sorted out, everything will be delivered.
• 00:41:43 The End Result: Code running highly optimizied then will never see it.
• 00:41:56 Kernel Signal: Non of what he is talking about is for Kernel or Hardware Interrupts.
• 00:43:59 Testing for sleep: There will be an indicator that you're trying to sync, and that will show as always bad.
• 00:44:57 Test Survey: Ask around how to debug race conditions.
• 00:45:00 Bad Testing Code: Sleep for a second will cause things to get caught.
• 00:45:19 Tool Library: Made library called Light Weight Logging.
• 00:45:57 Print Takes Mutex: A bad culprit and print has bad culprit for HEISEN bug.
• 00:47:57 L3 - Tool: A tool called L3 which has a L3 log and D in It L3 more or less same code.
• 00:49:29 Result: On one thread on local host has one nano second which Lipc has hundred nanocs.
• 00:50:39 Fast Tool: Writes is array memory mapped in the file.
• 00:51:46 Tool Does: Sync and add fetch that does atomic increment.
• 00:52:41 Linux magic - Tool: Has magic Global one that looks at one the source code to make sure code can create a thread.
• 00:54:14 New Tool: Writes are for assembly, so not smarter than the assembly for it so will use that one.
• 00:56:45 Tool: Thread fuzzing make race more likely.