The topic of this transcript falls under Software Engineering, specifically focusing on C++ Library Design, Application Binary Interface (ABI) Stability, and Source Compatibility (SC). A suitable group of people to review this topic would be:
Senior C++ Library Architects and Developers: Those responsible for designing, implementing, and maintaining large-scale C++ libraries (e.g., operating system components, frameworks like Qt, scientific computing libraries).
Compiler Developers and ABI Experts: Individuals working on C++ compilers or those with deep knowledge of platform-specific ABIs (e.g., Itanium ABI, Microsoft Windows ABI).
Technical Leads and Project Managers in projects with long-term C++ library maintenance requirements.
Abstract:
This presentation by Marc Mutz from Meeting C++ 2025 provides an in-depth exploration of Binary Compatibility (BC) and Source Compatibility (SC) in C++ user libraries, drawing heavily on decades of experience from the KDE and Qt projects. The talk meticulously defines BC, SC, Application Binary Interface (ABI), and Application Programming Interface (API), emphasizing their distinct natures and the ability to manage them independently. Key challenges related to name mangling, exported symbols, and structural changes (e.g., to enums or virtual functions) are detailed. Mutz introduces and elaborates on various techniques for maintaining BC, such as PIMPL and reserved fields, before presenting a novel "cheating" mechanism—the removed_API.cpp file—that allows for API evolution (e.g., function replacement) without breaking ABI, thereby enabling a "user-configurable rolling binary compatibility window." The discussion also highlights the persistent challenges posed by wholesale exported classes and virtual functions in ABI evolution.
Summary: Binary Compatibility 100 - Marc Mutz - Meeting C++ 2025
0:00 Introduction & Speaker Background: Marc Mutz, a Principal Software Engineer at The Qt Company, introduces the talk on binary compatibility in C++. He emphasizes his role in API reviews at Qt and acknowledges that the talk reflects accumulated experience and "hearsay" rather than definitive academic expertise, necessitating some omissions for brevity.
1:28 Scope & Exclusions: The talk focuses on dynamic linking of C++ user libraries. It intentionally excludes C++20 modules (which solve some but not all BC problems), linker scripts for symbol versioning, static linking (a "completely different beast" due to no runtime compatibility needs), and combining libraries compiled with different compiler vendors, versions, or flags.
2:34 Motivating Example: A function scan(int fd) is introduced.
Adding a defaulted argument options (scan(int fd, Options options = {})) is shown to be binary incompatible (mangled name changes).
Overloading with a new scan(int fd, Options options) is binary compatible but can be source incompatible if code takes the address of scan (resulting in an overload set rather than a single function).
The goal is to replace the function in the API while maintaining the old version in the ABI.
KDE & Qt Context: Both projects promise and maintain BC and SC within major releases, accumulating significant experience in the field.
Binary Compatibility (BC) (8:20): A library is BC if a program dynamically linked to an older version continues to run with newer versions without recompilation. This applies to libraries, not fully linked programs, and focuses on "still links" rather than full behavioral compatibility. BC is a local property of types and functions.
Source Compatibility (SC) (12:06): A library is SC if a program needs recompilation but no source code modifications to run with a new version.
BC != SC (12:54): The talk stresses that BC and SC are distinct and can be independently varied (e.g., adding a default argument is SC but not BC; adding an overload is BC but can be SC).
Application Binary Interface (ABI) (13:35): What the linker sees (mangled names, exported symbols), sitting below C++.
Application Programming Interface (API) (13:59): What the compiler sees (C++ standard constructs, overloading, default arguments, typedefs).
Backwards Compatibility (14:40): Code compiled against an old library runs against a new library. Qt provides this in major releases.
Forwards Compatibility (15:39): Code compiled against a new library runs against an old library. Qt provides this in minor releases (e.g., 5.15.x against 5.15.0).
The process of creating unique names for overloaded C++ functions at the ABI level.
Encodes name, scope, signature (arguments, template args, CV/L-R value qualifiers).
Does not encode argument names, default values, top-level CV qualifiers (e.g., const int vs int), noexcept (mostly), or access specifiers (except on Windows). Return type is also not encoded (except on Windows).
Changes to mangling inputs are an ABI break; changes to non-inputs should be BC (platform-dependent).
Unix by default exports all external functions; Windows requires explicit __declspec(dllexport) directives.
Exporting is the process of reducing (Unix) or extending (Windows) the set of callable symbols.
Classes are often "wholesale exported" (e.g., Q_CORE_EXPORT class MyClass {}), which recursively exports all members, static data, and virtual tables.
25:06 Exporting Best Practices: Avoid exporting inline symbols (to keep them out of the ABI), selectively export extern symbols, and avoid inline Vtables (define destructors out of line for polymorphic classes to pin the Vtable to a single TU). Do not wholesale export non-polymorphic classes; export individual functions/variables only.
27:00 Unexported Inline Symbols: Linkers deduplicate inline code only per executable. In dynamic linking, a shared library and an application are two executables, meaning inline functions/variables can have duplicate copies, potentially leading to multiple singleton instances if not carefully managed.
28:56 KDE Binary Compatibility Page: The speaker references the KDE Wiki page as a key resource, detailing pitfalls such as adding enumerators changing an enum's underlying type, covariant return types affecting pointer values in virtual functions, and issues with multiple inheritance.
PIMPL (Pointer to Implementation) Idiom: Isolates implementation details, preventing ABI breaks from internal changes.
Reserved Fields: Explicitly reserving void* pointers, bits in bitfields, or making padding explicit to allow for future data additions without changing class size.
32:54 Move Semantics with PIMPL: Inline move constructors/assignment operators are possible with PIMPL by using swap or std::exchange, provided null pointer semantics are defined (e.g., partially formed state). Challenges exist with std::unique_ptr and inline move constructors due to compiler behavior (calling data member destructors).
38:39 Virtual Hooks: Adding a single virtual hook function to a class hierarchy to "replace" N virtual functions, allowing for future extensions without adding new virtual functions to the Vtable (which is an ABI break). Requires foresight.
Inline Namespaces: (Briefly mentioned, skipped for time)
Types don't exist in the ABI in the same way as in the API; ABI is about memory layout and functions acting on it.
ABI is maintained as long as memory layout doesn't change and existing functions aren't removed. Adding fields at the end of a type is generally an ABI break due to stack allocation assumptions.
43:37 The "Cheating" Mechanism (removed_API.cpp): A novel technique to replace an API function while retaining its ABI:
Define a preprocessor macro (e.g., MAGIC_REMOVED_API_ONLY) that, when set to 1, makes the old function declaration visible.
In a dedicated removed_API.cpp translation unit, set this macro to 1, include the header, and provide an implementation for the old function (e.g., calling the new function after casting).
For all other translation units (including client code), the macro is 0 (or undefined), making only the new function visible in the API.
This allows the linker to find the old function (maintaining ABI) while the compiler only sees the new function (evolving API).
This mechanism is versioned in Qt (Q_CORE_REMOVED_SINCE) and is cross-platform. It "works post-hoc," meaning it can be applied after an ABI problem is discovered, and automatically cleans up old API functions in future versions.
48:23 User-Configurable Rolling Binary Compatibility Window: Qt leverages this mechanism to allow users to set a configurable cutoff version for ABI zombies, effectively creating a rolling BC window.
48:45 Drawbacks & Impediments: This "cheating" mechanism does not work for virtual functions, as adding or replacing virtual functions fundamentally alters the vtable layout, which is an ABI break. This is a major impediment for ABI evolution in frameworks like Qt.
Vtable Guarantees: ABI definitions for vtable layout are platform/compiler-specific (e.g., Itanium ABI for GCC, MSVC ABI for Windows). There are no C++ standard guarantees across different compilers or even versions that vtables will be compatible. Adding a virtual function to a base class in a dynamically linked hierarchy is an ABI break, as it changes vtable offsets.
Automated Testing: Tools exist (scripts) to perform binary diffs between shared libraries, showing added/removed functions and potentially flagging enum/type size changes. While helpful for flagging, they cannot definitively determine behavioral BC issues.
Static Linking: Static linking lacks the "executable boundary" of dynamic linking. Linkers aggressively remove unused code and merge duplicates of inline functions within a single executable, eliminating the need for dynamic BC but making certain ABI management tricks (like the removed_API.cpp mechanism for inlining functions) difficult or impossible without ODR violations.