This technical presentation outlines BYD’s architectural migration from Airflow to a multi-cluster Argo Workflows environment to support the extreme scaling requirements of autonomous driving data pipelines. Processing over 1PB of data daily across 3,000+ GPUs, BYD faced significant bottlenecks with Airflow’s state synchronization and scalability. The new Kubernetes-native solution leverages Argo Workflows for high-level orchestration and Ray clusters for distributed GPU computing, achieving a million-task daily throughput. Key optimizations include custom informer cache mechanisms to resolve update delays, offloading event processing to reduce API server pressure by 50%, and implementing hierarchical namespace-level concurrency controls. The transition resulted in an 11x increase in execution speed and a 30% reduction in total computing costs while maintaining a 99% success rate across 40,000 concurrent workflows.
Empowering Autonomy: BYD's Million-Task Scaling with Argo Workflows
0:31 Team Introduction: Jumbo and Winang (BYD) lead autonomous driving engineering focusing on automatic annotation; Shuangkun Tian (Alibaba Cloud) is an Argo Workflows maintainer specializing in large-scale data orchestration.
1:48 The Scale of the Challenge: Automatic annotation for autonomous driving requires processing at least 1PB of multi-sensor data per day to generate model training sets.
3:20 Limitations of Airflow: BYD migrated from Airflow due to severe scalability bottlenecks. Frequent state synchronization caused tasks to "hang" even after completion, and the system lacked native GitOps support and immutable versioning for pipelines.
5:39 Multi-Cluster Argo Architecture: To surpass single-cluster Kubernetes limits, BYD implemented a multi-cluster topology managed via Argo CD and Alibaba Cloud dashboards. This ensures identical, version-controlled environments across all clusters.
7:15 Hybrid Resource Management: The system utilizes Alibaba’s proprietary PPU (AI chips) for GPU workloads and a mix of ECS (Elastic Compute Service) and elastic instances for cost-effective CPU scaling during burst periods.
10:45 Integrating Ray for GPU Optimization: While Argo manages the end-to-end lifecycle, GPU-intensive tasks are offloaded to Ray clusters. This hybrid approach utilizes Ray’s superior distributed computing for model execution while relying on Argo’s robust supervision and retry logic.
14:58 Concurrency and Quota Control: To prevent scheduler saturation, BYD employs namespace-level concurrency limits. High-priority tasks can "borrow" resources from lower-priority quotas during peaks, preventing resource starvation.
Informer Cache Overhaul: Developed a custom cache to resolve "informer update delays," ensuring the controller uses the latest resource versions and preventing redundant pod creation.
Control Plane Relief: Optimized "patch" and "create" requests from the user side, reducing central API server CPU utilization by 50%.
Event Offloading: Shifted time-consuming operations (like listing/deleting pods) out of the main event handler to prevent Workflow Controller Out-of-Memory (OOM) errors.
21:58 Performance Metrics: The system supports a pending queue of 200,000 workflows and handles 20,000 to 40,000 concurrent active workflows with scheduling latencies as low as 50ms.
Speed & Efficiency: Task execution is 11x faster than the legacy system.
Cost Reduction: Improved resource utilization led to a 30% saving in total infrastructure costs.
Community Impact: Performance fixes regarding informer bottlenecks and controller stability have been upstreamed to the Argo Workflows open-source project.