Abstract:

Binlog is a high-performance C++ library designed for structured binary logging. It addresses the common inefficiencies of traditional text-based logging, such as performance overhead from string conversion, data redundancy, loss of precision, and difficulties in automated parsing. The library's core principle is to serialize log arguments in their native binary format, storing static information like format strings and source code locations only once. This results in extremely fast, compact, and precise log files. The library features an asynchronous, lock-free architecture to minimize application latency. The generated binary logs are not human-readable but are processed offline by the bread utility, which converts them into customizable, sorted text output. To prevent data loss in case of a crash, a brecovery tool is provided to salvage unflushed log entries from application coredumps.

An Overview of Binlog: The High-Performance C++ Logging Library

• Core Concept: Structured Binary Logging: Binlog moves away from inefficient text logs by writing data in a compact binary format. Static information (e.g., the format string "Processing request #{}") is stored only once, while dynamic arguments (the request number) are appended in their native form. This drastically reduces file size and avoids costly string conversions on the application's hot path.
• High Performance & Asynchronous Design: The library achieves logging speeds in the tens of nanoseconds by using asynchronous, lock-free queues. Application threads write log events to an in-memory buffer without blocking, and a separate consumer process writes the buffered data to disk, minimizing I/O latency.
• Data Integrity and Crash Recovery: Unlike text logs that lose numeric precision, Binlog preserves the exact binary representation of logged values. It includes a brecovery tool that can scan an application's coredump to extract and save log events that were still in memory buffers at the time of a crash.
• Log Processing with bread: Since the log files are binary, the bread command-line utility is essential for viewing them. It converts the binary stream into human-readable text, allows for fully customizable output formats, and can sort all log entries by timestamp, which is crucial for analyzing multi-threaded applications.
• Extensive Type Support: Binlog natively supports logging a wide range of C++ types, including primitives, containers (std::vector, std::map), strings, smart pointers, and tuples. Custom user-defined structures and enums can easily be made loggable using simple adaptation macros like BINLOG_ADAPT_STRUCT.
• Advanced Filtering and Organization: The library offers powerful features for managing log output:
  • Severity Control: Log verbosity can be adjusted at runtime by setting a minimum severity level (e.g., INFO, WARNING, ERROR).
  • Categories: Log events can be assigned to different categories to logically separate output from various components of an application.
  • Named Writers: To clarify log origins in multi-threaded contexts, writers can be assigned names (e.g., "WorkerThread1") that appear in the final text output.
• Flexible Output: While binary logging is the default, Binlog provides a TextOutputStream to write human-readable text directly from the application. It also supports multiplexing, allowing a single log stream to be routed to multiple destinations, such as a binary file for archival and a text-based console for real-time error monitoring.