This talk by Vitali Speedu, a machine learning scientist at Adyen, addresses a frequently overlooked component of observability and anomaly detection pipelines: data aggregation. He argues that the common practice of using a fixed, hierarchical partitioning of data (e.g., by company, payment method, country) creates significant problems at scale. These issues include a combinatorial explosion of time series, severe data skew where most series are too sparse to monitor, low overall data coverage, and an unfair distribution of monitoring resources across different business units.
To solve this, he proposes a shift from fixed to optimal partitioning. The goal is to create partitions that maximize the global "problem detection rate." The core of the solution is a novel metric based on the probability of detecting a significant drop in a given time series, which uniquely considers both the rate and the raw volume of transactions. This metric guides a scalable, linear-time agglomerative clustering algorithm that intelligently groups smaller data segments. The algorithm first operates on a fixed variable hierarchy and then optimizes the order of that hierarchy. This method successfully reduced 7 million sparse time series to 50,000 robust and monitorable ones, achieving 100% data coverage and discovering more alerts, despite a massive reduction in the number of series.
Optimal Observability: Re-thinking Data Aggregation for Anomaly Detection
0:00:39 - The Overlooked Problem: The initial step in monitoring pipelines—aggregating individual data points (like transactions) into time series—is often taken for granted but is critical for effective anomaly detection.
0:05:09 - The Flaw of Fixed Partitioning: Using a predefined, fixed hierarchy to slice data leads to major issues:
Curse of Dimensionality: A small number of dimensions (e.g., company, payment method, country) can create a massive number of time series (e.g., 7 million).
Data Skew & Low Coverage: Most of these time series are too sparse to be statistically significant, forcing teams to discard them. This results in incomplete monitoring coverage, where as little as 2% of the data might be contained in 98% of the time series.
Unfair Treatment: Smaller business units or partners may receive little to no monitoring coverage compared to larger ones.
0:10:41 - The Goal: Smart Partitioning: The proposed solution is to move away from fixed hierarchies to an optimal partitioning of the entire data space. The key objective is to create partitions that globally maximize the discovery rate of real-world problems and ensure every single transaction is part of a monitored time series.
0:15:36 - A New Metric for Detectability: To create smart partitions, a new metric was developed to quantify the "detectability" of a problem. It calculates the probability of detecting a performance drop of a specific size (e.g., d fewer approved transactions). Unlike standard metrics like entropy, this metric is sensitive to the raw counts (volume) of the data, not just the rate, allowing it to better differentiate between time series.
0:23:34 - A Scalable Agglomeration Algorithm: To partition the data without the process being computationally prohibitive (NP-hard), a scalable, linear-time algorithm was created.
A variable hierarchy is defined (e.g., split by company, then payment method).
At each level, partitions that are too small are merged using the new detectability metric. To remain scalable, this uses an approximation technique involving sorting and a moving window.
Finally, the algorithm optimizes the order of the variables in the hierarchy itself to find the best global result.
0:27:02 - Key Results: The new approach yielded significant improvements:
Drastic Reduction: Reduced 7 million time series to a controllable 50,000.
100% Coverage: Every transaction is now monitored, eliminating blind spots.
Homogeneous & Robust Series: The resulting time series are all "beefy" enough for reliable analysis and are distributed fairly across business units.
More Alerts Discovered: Despite monitoring far fewer series, the system found more actual problems.
0:28:58 - Challenges and Considerations: The solution is not without its trade-offs. The resulting partitions are more complex and harder for end-users to interpret, implementation is non-trivial (requiring custom UDFs), and the system assumes that the data used for training the partitions is "healthy" and not already broken.