Today, Andrew is taking us on a deep dive into the rapidly developing world of AI weather models and their blind spots, which may make them ill-suited to a warming world, at least for now. —Angela Fritz, Meteorologist

From Andrew Freedman

A man takes photos of boats that were damaged by Hurricane Ian in Fort Myers, Florida, in September 2022. (Giorgio Viera/AFP/Getty Images)

Meteorology is going through an artificial intelligence-induced disruption, as new AI-driven computer models are beating the accuracy of some of the tried-and-true, physics-based systems that forecasters have been using for decades.

Meteorologists have been quick to adopt these AI models in their forecasting, as tech companies such as Google, Nvidia and Huawei, among others, unveil increasingly sophisticated models that can be run in minutes on ordinary computers.  

You don’t have to look hard to understand why: the physics-based models must be run on massive supercomputers that take hours to spit out the latest projections.  

However, just as the models that are rooted in physical equations are only as good as the weather observations that go into them, the AI models are limited by the data they are trained on. Turns out, “garbage in, garbage out” applies to AI, too. 

This is because AI models such as Google DeepMind’s GraphCast and Huawei’s Pangu-Weather make predictions based on what has happened in the past, using decades of observational data to teach themselves how the weather evolves. Crucially, though, they don’t actually learn how the atmosphere works on a physical level. 

There’s no question the AI modeling developments in the past few years are exciting, and even have contributed to major forecast successes, including the prediction of Hurricane Melissa’s rapid intensification last year. That was in part based on Google DeepMind’s tropical weather model, which is AI-driven.  

These new models can even perform well in predicting certain extreme events, largely in cases where similar events exist in the training data.  

But what if a major, record-breaking event happens, which by definition has not happened before? Will the AI models predict it accurately, or swing and miss?

This is a question I’ve asked many in the AI weather space during the past couple of years, and now a group of weather researchers have closely examined this question about so-called “gray swan” events in a newly published study in the journal Science Advances. Gray swans are record-breaking extremes absent from models' training data.  

A man points at vehicles stranded in floodwater on the Major Deegan Expressway in the Bronx borough of New York following Hurricane Ida in October 2021. (Craig Ruttle/AP)

The study compares the skill of the most accurate physics-based model, which is the high-resolution Euro, to some of the top-of-the-line AI models. This is an increasingly important investigation as climate change amps up the severity and frequency of certain types of record-breaking extremes, such as heat waves and rapidly intensifying hurricanes.

They found the physics-based Euro model beats the AI models when forecasting record-breaking heat, cold and wind across nearly all timescales. They also concluded that the five AI models they tested tended to underestimate both the frequency and intensity of record-breaking events.  

That limits how much we can rely on them to anticipate some of the unprecedented extremes now occurring in this rapidly warming world, and serves as a caution against incorporating these AI models into operational early warning systems.  

“While other studies often look at moderately extreme events, we explicitly target the weak spot of AI models, namely where they do not have enough

training observations in historical data,” study co-author Sebastian Engelke of the University of Geneva told me. 

Researchers who were not involved in the new study said its findings are important, though not entirely surprising. “This new study highlights an important limitation with major implications,” said Pedram Hassanzadeh of the University of Chicago.

One of the lessons of this study (and others) is that we should continue refining and using physics-based models while also improving and learning more about the capabilities and flaws of the AI models.

Another lesson is that models trained on low-resolution data can’t be expected to capture some of the high-resolution processes that happen in the most extreme events, Mike Pritchard, director of climate simulation research at NVIDIA, told me.  

Pritchard and his colleagues are working on models that use high-resolution training data to simulate weather conditions, and they have shown significant progress, he said. “What we should expect is based on the data the AI has the benefit of learning from.”

Researchers say there are other paths forward that could improve AI models’ abilities to predict the gray swans.  

The bottom line, to put it plainly, is that it would be a mistake to go all in on AI weather forecasting, at least for now. We need more research. As it stands, they can excel in routine weather conditions but are of more limited value in this increasingly extreme world.  

A fishing boat travels through the Hawizeh Marsh in Iraq’s Maysan province on Sunday, April 26. Rains have restored water to the marsh following a long period of drought. (Hussein Faleh/AFP/Getty Images)

Unlock deeper analysis and exclusive videos on the stories you care about. Subscribe here. 

Let us know. We love to hear from you!