Artificial intelligence (AI) can generate remarkably convincing text. But when it comes to something as complex and impactful as weather forecasting, sounding right isn't the same as being right.
A new study led by Environmental Institute Climate Fellow Timothy Higgins introduces a new way to test whether AI models can actually understand and accurately describe the weather patterns represented in numerical forecasts.
Published in Environmental Data Science, the study introduces SynopticBench, a dataset containing more than 1.3 million examples pairing weather forecast data with detailed discussions produced by the National Weather Service (NWS). The researchers also developed a new evaluation framework, called Synoptic Phenomena Alignment and Coverage Evaluation (SPACE), to determine whether AI-generated forecast discussions identify the right weather phenomena in the right locations.
“AI models are becoming increasingly good at generating language that sounds like a human wrote it,” said Higgins. “But for weather forecasting, it's not enough for the language to sound plausible. The model needs to identify the right atmospheric features and put them in the right place.”
Human weather forecasters synthesize an enormous amount of information about the atmosphere to explain what is happening and what is expected to happen next. Higgins and his collaborators, Antonios Mamalakis and Chirag Agarwal with UVA’s School of Data Science, recognized that what those forecasters created provide an unusually valuable resource for training and testing AI models.
To the researchers' knowledge, SynopticBench is the first large-scale benchmark specifically designed to generate written discussions from images of numerical weather forecasts. One of the biggest challenges in evaluating AI-generated weather discussions is that traditional language metrics can miss important errors.
For example, an AI-generated forecast might use many of the same words as a human-written forecast and therefore score well on conventional measures of text similarity. But if AI describes a high-pressure system as a low-pressure system (or puts a weather system in the wrong part of the country), the forecast is fundamentally wrong.
To address that problem, the researchers developed SPACE. Rather than simply asking whether the AI's words resemble those of a human forecaster, SPACE evaluates whether the model identifies important weather phenomena and associates them with the correct geographic locations.
The results highlight the significant limitations of current AI systems.
None of the models consistently outperformed a simple climatology-based approach when evaluated across the entire forecast domain. In other words, AI models still struggle to accurately describe all of the important weather systems occurring across a large geographic region.
The researchers also found that correctly identifying whether a system was high or low pressure was easier for the models than identifying where that system was affecting the forecast area.
And in some cases, conventional language metrics failed to distinguish between good and bad forecasts. One example involved an AI model that incorrectly described a trough over the southeastern United States when the actual forecast showed a ridge. Despite the substantive error, traditional text metrics did not clearly identify the generated discussion as worse. SPACE, by contrast, assigned the forecast a score of zero for the pressure-system description.
The findings underscore an important challenge as AI moves into applications where accuracy matters.
“Before we can trust AI to help communicate weather forecasts, we need better ways to measure what these models actually understand,” Higgins said. “SPACE gives us a way to move beyond whether an AI-generated forecast sounds good and start asking whether it is physically and geographically correct.”
The framework could also be expanded to evaluate other phenomena, geographic regions, and time scales. Future AI systems could incorporate additional atmospheric variables, longer forecast horizons, and even video representations of how weather systems evolve.
The researchers emphasize, however, that AI-generated forecast discussions are not yet a replacement for the detailed analyses produced by human meteorologists.
“Weather forecasts are incredibly complex,” Higgins said. “We're starting with individual pieces of the forecast so that we can rigorously test whether these models understand those pieces. That's an important step toward building AI systems that can eventually reason about the atmosphere in a more complete way.”