Binlog is a high-performance C++ log library designed to overcome the limitations of traditional text-based logging by producing structured binary logs. It addresses common issues such as redundancy, wasted space, loss of precision, high production cost, and parsing difficulties inherent in conventional logging solutions. By serializing log arguments directly and storing static metadata only once, Binlog enables significantly faster log event creation (tens of nanoseconds), results in smaller and more precise log files, and facilitates easier machine-readable data consumption. The library supports asynchronous, lock-free logging, offers extensive type serialization, and provides auxiliary tools: bread for configurable text conversion (including time-based sorting) and brecovery for recovering buffered log events from application coredumps.
Summarizing Binlog: A High-Performance Structured Binary Logging Library
Binlog Overview:
A high-performance C++ log library (BINLOG_INFO("Log anything! {}, {} or even {}", ...)).
Produces structured binary logs, addressing issues with traditional text logging.
Actively developed and tested ("live at head") on Linux, macOS, and Windows with C++14.
Motivation for Binlog (Problems with Traditional Logging):
Redundancy: Severity, format strings, and file paths are often printed multiple times.
Wasted Space: Timestamps and values are inefficiently represented in text (e.g., 29 bytes for a timestamp vs. 8 bytes).
Loss of Precision: Original data (e.g., float values) can lose precision during text conversion.
High Production Cost: String conversions (timestamps, float values) are expensive on the application's hot path.
Parsing Difficulty: Not trivial for automated tools to parse reliably.
Synchronous vs. Asynchronous Trade-offs: Synchronous logging can be slow due to locking, while asynchronous logging may produce unsorted output and risk data loss on crashes.
Binlog's Solution: Structured Binary Logs:
Efficiency: Static parts (severity, format string, file/line) are saved only once. Log arguments are copied as-is (binary), not converted to text.
Key Benefits: Leads to less redundant, smaller, and more precise logfiles. Significantly faster to produce, saving CPU cycles.
Structured Nature: Binary logfiles are easily consumed by other programs, avoiding fragile text parsing.
Crash Recovery: The brecovery tool can recover buffered log events from coredumps, addressing data loss in asynchronous logging.
Core Features:
Versatile Logging: Supports logging a wide array of types including fundamentals (char, bool, integers, floats), standard containers (vector, map, set), strings (const char*, std::string, string_view), pointers (raw, unique_ptr, shared_ptr, optionals), pairs, tuples, enums, and custom user-defined structures (when adapted).
High Performance: Benchmarked to create log events in "tens of nanoseconds" (e.g., 8-34ns for an integer, 30ns for three floats, 38ns for a string on an Intel Xeon E5-2698).
Asynchronous & Lock-free: Utilizes lock-free queues for minimal overhead on the application's hot path.
Configurable Output: Uses {} placeholders in format strings; supports vertical separation via custom categories and horizontal separation via runtime-configurable severities.
Log Sorting: Logs can be efficiently sorted by time during reading using the bread tool.
Included Tools:
bread (Binary Reader): Converts binary logfiles (.blog) to human-readable text. It allows flexible output formatting (e.g., custom timestamps and fields), sorting events by timestamp (-s), and can process piped or live log streams.
brecovery (Binary Recovery): Recovers unconsumed log data from application coredumps, saving it into a new binary logfile that can be read by bread.
Advanced Usage & Customization:
Session & SessionWriter: Explicit objects for managing log streams and writers, enabling concurrent writing from multiple threads and custom queue capacities.
Named Writers & Categories: Writers can be named for better identification in text output; events can be assigned categories for component-based filtering.
Runtime Severity Control:Session::setMinSeverity() allows dynamic adjustment of logging verbosity, discarding lower-severity events without evaluating their arguments.
Log Rotation & Multiple Outputs: Supports log rotation by changing the output stream for session.consume(), requiring reconsumeMetadata() for new self-contained files. Can multiplex log streams (e.g., full binary log to file, high-severity text to stderr) using TextOutputStream and EventFilter.
Limitations & Considerations:
Global Destructor Context: Logging from global object destructors using default macros (which access global state) may lead to undefined behavior; explicit SessionWriter usage is recommended in such scenarios.
Timestamp Assumptions: Assumes std::chrono::system_clock measures UTC time without leap seconds, consistent with C++20 standards.