This presentation details Linfeng's summer internship project on accelerating the thermodynamics module, particularly the Convective Available Potential Energy (CAPE) function, within the MetPy Python library by integrating C++ using Pybind11. The core motivation is to improve the performance of computationally intensive meteorological calculations, which involve multi-stage processes and large datasets. Linfeng discusses the design challenges encountered, such as handling units, input validation, array broadcasting, and integrating Python's user-facing logic with C++'s core computational efficiency. While significant performance improvements (up to 90% reduction in runtime for some functions) were achieved for many thermodynamics functions, challenges persist with multi-dimensional array handling in functions like dry_lapse. The presentation also briefly covers a secondary project on visualizing Jupiter's atmospheric simulations in the Integrated Data Viewer (IDV), showcasing complex structures like vortices and plumes, and discussing the visualization challenges presented by planetary scales.
Accelerating MetPy Thermodynamics with C++ and Visualizing Jupiter's Atmosphere
0:00 Project Introduction: Linfeng introduces his summer internship project: accelerating the thermodynamics module in MetPy using C++ to improve performance, especially for the complex Convective Available Potential Energy (CAPE) function.
0:47 CAPE Calculation Complexity: The CAPE calculation is a multi-stage process involving several thermodynamic building blocks like LCL (Lifting Condensation Level), Dry/Moist Lapse profiles, finding intersections (FLC, EL), and area integration. These steps are computationally intense for large datasets in Python.
2:09 Pybind11 as the Tool: Pybind11, a lightweight C++ library easily exposed to Python, is the main tool for creating a bridge between C++ and Python for performance optimization.
3:07 Implementation Challenges: Key challenges included handling units in C++, validating input (scalar vs. array), broadcasting arrays, incorporating existing Python constants, and assembling complex return values.
4:50 Python-C++ Design Philosophy: A layered approach was adopted: Python handles user-facing logic and complex data assembly (e.g., determining liquid vs. solid phase in saturation vapor pressure), while C++ focuses on core, specialized computations for speed.
7:44 Vectorization Strategy: C++ functions are written as scalar functions (single value input/output). Python-level API vectorization (using py::vectorize) handles array inputs, with looping occurring efficiently on the C++ side. Internal C++ vectorization also occurs when higher-level C++ functions call these scalar functions.
10:00 LCL Function Specifics: The py::vectorize utility has limitations when functions return multiple values (e.g., LCL returns both pressure and temperature), requiring manual member function writing for vectorization.
11:00 CAPE Architecture & Dependencies: The CAPE function's architecture involves a deep, intertwined tree of interactions and dependencies. Many bottom-level C++ functions (like specific humidity, dew point) are exposed to Python.
11:49 Performance Benchmarking with ASV: Using ASV (a Python benchmarking tool), significant runtime reductions were observed for functions like EL, LFC, and partial profile, with some reaching over 50% and moist lapse calculation for 1D arrays showing a 90% reduction.
12:37 Dry Lapse Function Issue: The dry_lapse function showed slower performance (up to 15 times longer) for 2D/3D array inputs when broadcasted from Python to C++, implying Python handles its own broadcasted arrays more efficiently. For now, the Python version of dry_lapse might still be preferred by users.
14:27 Future Work for MetPy: Future improvements include optimizing math functions (trapezoidal integration, find intersection), resolving the dry_lapse issue, completing the partial_profile function, and moving more surrounding functions into C++ for further speedup.
16:55 Jupiter Simulation Visualization (Secondary Project): Linfeng also worked on visualizing Jupiter simulations in the Integrated Data Viewer (IDV) with fine-grained resolution, deep pressure levels (1-88 bars), and a 50-degree latitude/longitude range.
17:46 Jupiter Atmospheric Phenomena: Visualizations revealed clear vortices, plumes traveling east to west, and characteristic band and zone structures. Vortices extend deeply from shallow to deep pressure levels (0.1 bar to 85 bar).
20:02 IDV Potential & Challenges: IDV shows potential for visualizing other planets despite challenges with data format, stretching simulations to align with Earth-sized globes, and scaling issues due to Jupiter's size.
21:38 Jupiter Wind Speeds & Circulation: Jupiter's jet streams can reach 100-200 km/h. Its circulation is driven by instability caused by water and ammonia vaporization and condensation, triggering vortices.
27:14 Key Learning from Internship: Linfeng's main learning was extensive C++ coding, which will be valuable for future simulation and dynamic modeling projects.
28:22 Python-C++ Interoperability: The combination of Python and C++ (via Pybind11) is a common and robust practice in scientific Python, used by libraries like Matplotlib and SciPy, particularly for speeding up tight computational loops that are slow in pure Python/NumPy.
30:08 Motivation for C++ Optimization: The decision to use C++ was driven by identifying CAPE as a high-value, highly-used function that hits known limitations of Python and NumPy (explicit loops, integral calculations) causing runtime issues (e.g., taking hours for large datasets).