In this talk, delivered upon receiving the 2024 Russell L. DeSouza Award, Jim Steenburgh of the University of Utah traces the parallel evolution of his career, numerical weather prediction, and the NSF Unidata program over the past 40 years. Starting with the poor-quality forecasts of the 1980s, Steenburgh highlights Unidata's crucial role in democratizing access to weather data for universities. He recounts the development and impact of foundational Unidata technologies like the Local Data Manager (LDM), netCDF, and CONDUIT, which enabled university-led real-time modeling and research. The talk transitions to the modern era, discussing the adoption of Python, MetPy, and visualization tools like the IDV. Looking forward, Steenburgh identifies the current period as one of profound disruption, driven by advances in Artificial Intelligence (AI), the growing role of the private sector, and significant pressures on higher education. He concludes by emphasizing the need for the academic community to adapt and innovate in this rapidly changing landscape.
A 40-Year Journey with Unidata: From Primitive Forecasts to the AI Revolution
0:09 The Early Days of Forecasting: Jim Steenburgh begins by describing the poor state of weather forecasting in the early 1980s, which featured low-resolution models (like the 190km LFM) and extremely limited public access to data, motivating his interest in the field.
1:46 The Birth of Unidata: In 1983, a workshop at the University of Wisconsin coined the term "Unidata" (University + Data). The program was officially formed in 1984 with the goal of providing universities with access to weather data, images, and interactive computing capabilities.
3:53 A Stable "Grand Design": By 1985, Unidata had established its core architecture of combining global data access with local interactive processing. Steenburgh notes this design has remained remarkably successful and relevant for 40 years.
4:48 Foundational Technologies Emerge: By the late 1980s, Unidata was testing prototypes of its most enduring technologies, including the Local Data Manager (LDM) for data distribution and the Common Data Format (CDF), the predecessor to netCDF.
6:00 Honoring Russell L. DeSouza: Steenburgh pays tribute to the award's namesake, Russell L. DeSouza, for his critical role in representing the interests of small colleges and universities within the Unidata community, ensuring broad benefits beyond large research institutions.
8:23 CONDUIT: A Solution for Higher Resolution Data: As universities began running their own real-time mesoscale models in the 1990s, they needed better access to initial model grids. Unidata responded by creating the CONDUIT project to deliver higher-resolution NCEP data to the community.
8:56 A Real-World Test: The 2002 Olympics: Steenburgh's team at the University of Utah used a suite of Unidata tools (LDM, netCDF, CONDUIT) to provide high-resolution weather support and machine learning-based forecasts for the 2002 Winter Olympics.
10:51 Democratizing Radar Data: The CRAFT project, championed by the late Linda Miller, was a major Unidata initiative to provide the community with real-time access to Level II radar data, which was previously often lost and unarchived. This enabled new avenues of research, including Steenburgh's own work on lake-effect snow.
12:48 The Shift to Python and MetPy: The talk highlights the modern era of Unidata development, including the creation of the MetPy library in 2015. Steenburgh shows how his own group now relies heavily on Python and MetPy for data analysis and visualization.
13:58 A Future of Disruption: Steenburgh argues that the next 10 years will bring more dramatic change than the last 40, driven primarily by the disruptive power of AI/ML and the increasing role of the private sector (e.g., Google, Nvidia) in developing weather models.
23:01 Challenges and Opportunities in the AI Era: The discussion highlights that while traditional weather science was driven from within the domain, AI innovation is often coming from external computer science fields. This creates both challenges and opportunities for recruiting students and adapting university curricula.
42:18 Modern Technical Barriers: A significant modern challenge for researchers is not a lack of tools, but institutional cybersecurity. Centralized IT departments and strict security protocols can create major hurdles for installing and using specialized scientific software like the IDV in both classroom and field settings.
46:16 The Ultimate Goal: "Science at the Speed of Thought": When asked for a "blue sky" wish, Steenburgh describes his desire to interact with weather data conversationally, akin to Captain Kirk on Star Trek. He wants the ability to instantly visualize complex concepts (e.g., "show me wave numbers 0-5 at 500mb") without having to write code, a future he believes AI may help realize.