This presentation details Jaye's summer internship work at Unidata, focusing on implementing a robust performance benchmarking system for the MetPy Python library. The core motivation is to quantify the speed improvements from Linfeng's C++ification of MetPy functions. The system leverages open-source tools like ASV (Airspeed Velocity) for timing code, Jenkins for continuous integration, Docker for isolated and consistent execution environments, and GitHub Actions for automatic reporting.
Jaye demonstrates a live benchmark website that visualizes historical performance data for various MetPy functions, highlighting significant speed changes linked to specific code commits (e.g., LCL's analytic calculation for speed, saturation vapor pressure's increased accuracy/slower performance). The workflow automates weekly benchmarks on consistent hardware, pushing results to a dedicated repository for publication. Furthermore, a comparative benchmarking feature integrated into GitHub Pull Requests is introduced to automatically detect and prevent performance regressions before they are merged into the main codebase, setting a threshold for acceptable slowdown.
MetPy Performance Benchmarking: A Deep Dive into Automation and Analysis
0:05 Introduction & Motivation: Jaye presents on performative benchmarking of MetPy using ASV and a CI/CD workflow, primarily to quantify performance improvements from Linfeng's C++ optimizations.
0:51 Live Demo of ASV Results: The presentation showcases a web interface displaying historical performance benchmarks for numerous MetPy functions (e.g., apparent temperature, density, LCL).
0:42 Analyzing Performance Changes: The LCL function is highlighted, demonstrating a substantial speed increase due to a change from an iterative to an analytic calculation, directly verifiable on the benchmark graph.
4:19 Understanding Performance Trade-offs: The saturation vapor pressure function shows a deliberate 40% slowdown, an accepted trade-off for increased accuracy.
6:50 ASV (Airspeed Velocity) Tool: ASV is introduced as an open-source Python benchmarking package used by major libraries like SciPy, Astropy, and NumPy, capable of building performance history across code commits.
8:24 Consistent Hardware with Jenkins: Jenkins is utilized for CI/CD workflows, specifically for its ability to run benchmarks on consistent hardware (ML11 machine), which is crucial for reliable performance comparisons over time, unlike GitHub Actions' varied runners.
9:17 Isolated Environments with Docker: Docker containers are employed to run benchmarks in isolated environments, preventing interference with the host machine and ensuring portability across different operating systems (Linux, Windows, Mac).
11:01 GitHub Actions for Publication: GitHub Actions are used in conjunction with ASV to automatically generate and publish the user-friendly HTML benchmark reports to a static website.
11:51 Data Generation for Benchmarks: A consistent 4D artificial dataset is programmatically generated (data_array_generate.py) for benchmarking, as the full dataset is too large to store directly in the MetPy repository.
13:15 ASV Setup Methods: Benchmarks utilize setup_cache (for data parsing, runs once per file) and setup (for data slicing, runs before each individual benchmark) methods to efficiently prepare data for timing.
15:07 Automated Weekly Workflow: A detailed workflow is outlined, describing how Jenkins automatically triggers benchmark runs every Saturday morning, fetches relevant commits, runs ASV in Docker, and pushes updated results to the public repository for website deployment.
19:27 Comparative Benchmarking for PRs: A system for comparative benchmarking on pull requests (PRs) is implemented using ASV and GitHub Actions. It compares performance between branches and automatically fails PRs if a benchmark shows a significant slowdown (defaulting to 10%).
21:48 Local Comparative Benchmarking: Developers can run local comparisons before opening a PR, examining the performance impact of their changes (e.g., a C++ CAPE branch showing wet_bulb_temperature being 97% faster and dew_point 98% slower).
24:17 Unexpected Discoveries: Jaye noted that the MetPy codebase is highly interconnected, meaning changes to one function can unexpectedly impact the performance of many others, making performance isolation challenging.
28:28 Key Learnings & Skills: Jaye highlights the importance of good documentation and the development of strong programming problem-solving skills, including debugging and troubleshooting complex systems, as key takeaways from the internship.
30:52 Team Commendation: The presenter acknowledges the interns for their commitment to complex projects and delivery of significant results despite challenging start-of-summer circumstances.