This talk delves into advanced software testing strategies, moving beyond simple input-output assertions to address the complexities of real-world systems. The speaker, drawing on his extensive experience from HashiCorp and his current work on the Ghostty terminal emulator, introduces "snapshot testing" for verifying complex outputs like image rasterizations and resource graphs. He emphasizes the critical importance of "testability" by isolating side effects, transforming non-deterministic operations (like keyboard input encoding) into pure, testable functions. Furthermore, the talk explores novel approaches to testing GPU-rendered applications by separating CPU-side data preparation from GPU-side shader execution, using off-screen rendering and snapshot comparisons. Finally, it highlights the utility of virtual machines (specifically NixOS VMs) for creating reproducible, full end-to-end testing environments to simulate complex, external interactions, such as diverse input methods and desktop application behaviors, that are otherwise untestable.
Advanced Software Testing Strategies for Complex Systems
0:00 Introduction to Complex Testing: The talk aims to provide a tour of various testing situations, acknowledging that many real-world tests are not simple input/output assertions but involve complex states and side effects.
2:13 Speaker's Background: The presenter shares his experience from HashiCorp, where he worked on large-scale, security-sensitive software, and his current project, Ghostty, a GPU-rendered terminal emulator, highlighting how these diverse experiences shaped his testing philosophy.
6:36 Testability is Key: A central theme is the distinction between "how to test something" and "testability" (making code inherently easy to test). Both are crucial and must be paired.
7:58 Snapshot Testing: This technique involves capturing and committing complex output formats (e.g., images, structured data like resource graphs) as "snapshots" and then comparing future outputs against them. It's particularly useful for visual outputs or deeply nested data, providing better diffs for identifying regressions.
13:47 Isolating Side Effects ("Soap and Bubbles"): Considered the most important concept, this involves refactoring code to separate core logic (pure functions) from external I/O or complex system interactions. This transformation makes previously untestable code, like a terminal's key encoding logic, deterministic and easily testable with artificial inputs.
21:09 Testing GPUs and Compute: The speaker shares his pioneering efforts in testing GPU-rendered applications, noting a lack of public resources. His approach divides testing into CPU-side (data preparation, graph generation, tested with snapshots) and GPU-side (shader execution, tested by rendering to off-screen buffers and snapshot comparing pixel data).
34:30 VMs for End-to-End Testing: For highly complex and non-deterministic scenarios (e.g., network failures, hardware interactions, diverse input methods), virtual machines (specifically NixOS VMs) offer a powerful solution. They provide reproducible, isolated environments to simulate real-world conditions and test full application workflows.
37:00 NixOS VMs Framework: This framework allows defining entire virtual machines with specific software versions, simulating user interactions, and verifying outcomes through methods like screenshots or SSH. It's particularly effective for testing desktop applications and challenging multi-environment behaviors like Linux input methods.
43:05 Key Takeaways: The speaker concludes by summarizing the five main topics: the importance of isolating side effects, the utility of snapshot testing, innovative GPU testing methods, and the power of NixOS VMs for comprehensive, reproducible end-to-end testing.