Academics from the Alan Turing Institute and other institutions in the UK claim to have developed a model for weather prediction that can be trained on a desktop at a fraction the cost and time incurred currently by using supercomputers.
Based on work done at The Alan Turing Institute by Richard Turner, a professor of machine-learning at The University of Cambridge and the research lead for AI in weather prediction at The Alan Turing Institute. Scott Hosking is the interim director of science, innovation and technology at The Alan Turing Institute. They believe that Aardvark can replace the entire numerical forecasting pipeline .
A three-step pipeline is used to create the NWP. First, satellites, weather-balloons, ground stations and ships, aircraft and buoys are used to collect observational data. This is combined with a forecast from a recent period to estimate the state of the atmosphere. Second, the estimate is fed into an advanced computational model that simulates atmospheric dynamics to generate forecasts. Finaly, the raw forecasts are processed to correct biases and improve local accuracy. Human forecaster input is also incorporated. Other recent efforts to improve weather forecasting using machine learning, like Google DeepMindās GenCast have focused on the second step – the computational models – while Aardvark can replace all three steps.
According to an articlethat was published in the science magazine Nature this month, Aardvark is able to produce accurate forecasts orders of magnitude faster than existing systems. It does so without relying on NWP products during deployment. Generating a full forecast from observational data takes approximately one second on four NVIDIA A100 GPUs, compared to the approximately 1,000 node-hours required by Hres to perform data assimilation and forecasting alone, before accounting for downstream local models and processing.”
A quick video demo of the software is below.
“https://youtu.be/zAovQbs6jlw?feature=shared” Data-media=”x-videoplayer”>Youtube Video
It is claimed that the resulting forecast is as accurate as Americaās Global Forecast System (GFS). It relies on a tenth the amount of observational data that is used in traditional NWP system.
“We only used 10 percent of the data as weāre quite limited in terms of computational resources in my academic lab compared to, say, technology companies,” Turner said The Register. “With more data, I would expect the system to perform better. Aardvark is able to compete using only a fraction of the data partly because the AI models are trained directly to solve the task of interest (forecasting).”
- The secret for better weather forecasts could be a dash AI
- NASA and IBM have just opened sourced an AI model to fine-tune on your own
- Nvidiaās latest AI model targets severe weather
- This is the ugliest global warming chart you’ll need to see
- We humans brought the heat. Earth says that we are paying the price.
Aardvark needs some refinement, as it doesn’t yet have the same resolution as Europe’s Integrated Forecast System (IFS) . The researchers are working on it, but they also plan to add specialized modules that will focus on specific types like hurricanes, flooding, and other extreme weather. Aardvark is made up of three components, including an encoder, a processing unit, and a collection of decoding modules. The encoder is about 31 million parameters long and takes 13 training hours. The processor has about 54 million parameters. It requires eight hours of training using ERA5 a weather data collection, followed by three hours of fine tuning using the encoder output. Each decoder module has around 2 million parameters, and each takes 30 minutes to train. Aardvark, if you allow two more hours for fine-tuning from end-to-end, can be trained within 100 GPU hours.
After that, Aardvark will be able to create forecasts in a matter of minutes on a desktop. Google claims that its GenCast forecasting model can produce a fifteen-day forecast using a Google Cloud TPU in just eight minutes.
The source code for replicating the results in the Nature article is currently restricted, but the plan is to let Aardvark loose eventually. Turner said
“Everything will be open sourced when the print version of Aardvark goes live and anyone will be able to download it and train the system on tasks of interest,” . (r)
