Cloud Computing
July 3, 2022 1:10 AM Subscribe
I know it is popular to hate on things here but I can’t stand Wendover anymore. He totally glossed over the “complex models” portion of actual weather prediction. This was an over long video that came down to sending up sounding balloons is important as is radar.
posted by geoff. at 9:05 AM on July 3, 2022 [1 favorite]
posted by geoff. at 9:05 AM on July 3, 2022 [1 favorite]
One weather model researcher's opinion twenty years ago was that there wasn't much headroom left in model improvements, the need was for more measurements of current state (esp. over oceans), for the problem of "best estimate of conditions at time/place". People want to know whether the cloud will drop rain here or there; this fundamentally can't be predicted very far out no matter how smart the model, but knowing what's upwind in detail is critical. (Modeling local terrain features in enough resolution was also a gap, he said, but he left that to the hardware people to provide more compute power.)
But: plenty of work left in model research, he said, for the problem of estimating probability distributions: what is the likelihood of each possible amount of rainfall at a location? Or integrating this, what is the total rainfall over a watershed area? The models, he said, could really make you shake your head at what they thought was physically reasonable.
Just one guy at a conference twenty years ago. I'd be interested to hear from current workers.
posted by away for regrooving at 9:45 AM on July 3, 2022 [2 favorites]
But: plenty of work left in model research, he said, for the problem of estimating probability distributions: what is the likelihood of each possible amount of rainfall at a location? Or integrating this, what is the total rainfall over a watershed area? The models, he said, could really make you shake your head at what they thought was physically reasonable.
Just one guy at a conference twenty years ago. I'd be interested to hear from current workers.
posted by away for regrooving at 9:45 AM on July 3, 2022 [2 favorites]
Nicely informative, thanks Gyan!
Cool to know that weather forecasts were impacted at the beginning of the COVID pandemic because of the drop in airplane traffic. Also that the WMO is one of the few international organizations that operate outside of geopolitical considerations for the benefit of all.
posted by storybored at 8:47 PM on July 3, 2022
Cool to know that weather forecasts were impacted at the beginning of the COVID pandemic because of the drop in airplane traffic. Also that the WMO is one of the few international organizations that operate outside of geopolitical considerations for the benefit of all.
posted by storybored at 8:47 PM on July 3, 2022
He totally glossed over the “complex models” portion of actual weather prediction.
Former modelling team lead at a reinsurance company here, that was my thought too. A few things:
And yes, Irish people are obsessed with weather. On my phone I have the Apple one, Windy (where you can pick from ECMWF, GFS, ICON-EU, ICON, NEMS, and AROME), Met Éireann (for the rainfall radar), and Yr.no. But none of them can beat the prediction accuracy of an old Irish person who's been playing weather prediction on Ultra Violence mode (east coast) or Nightmare mode (west coast) all their lives.
posted by kersplunk at 12:23 AM on July 4, 2022 [18 favorites]
Former modelling team lead at a reinsurance company here, that was my thought too. A few things:
- The cells for modelling aren't 2D as shown in the video - they're 3D. You have data for cubes of air and water (pick your resolution!), you interpolate them to fill in the gaps, you can weight some measurements more than others, you have an algorithm for how they interact, then you press go and head for a coffee and another coffee.
- You model more from first principles rather than from historical data because we don't have enough historical data - for example for hurricanes we'd need satellite data, and we only started getting that in the mid-60's.
- The video implies that for ensemble runs, you're changing parameters. You're not - you're running Monte Carlo simulations and getting different numbers out of random number generators. It's typical to lock down the RNG seeds so you can compare across runs from one day to the next. I was always picky about seeding - I complained to my boss once when I found a hardcoded seed of 2468 in some of his throwaway code that wouldn't be run very often. I asked him why 2468 and he said "well, it's twice as good as 1234".
- The video gives the impression that the US has caught up with Europe. It may have in terms of compute power available, but the consensus is that ECMWF consistently outperforms GFS.
- All Monte Carlo simulations take the same time to run, because as soon as you make them faster, people just ask for more simulations.
- Private model suppliers like RMS and AIR are often used as sanity checks for runs, you can also buy predictions off-the-shelf.
- There's a difference between expected (this is the most likely outcome) and tail (this is the worst we think might happen). Modelling the tail of the distribution accurately needs more simulations to be run. It will also feed into say, what message a forecaster should communicate to the general public.
And yes, Irish people are obsessed with weather. On my phone I have the Apple one, Windy (where you can pick from ECMWF, GFS, ICON-EU, ICON, NEMS, and AROME), Met Éireann (for the rainfall radar), and Yr.no. But none of them can beat the prediction accuracy of an old Irish person who's been playing weather prediction on Ultra Violence mode (east coast) or Nightmare mode (west coast) all their lives.
posted by kersplunk at 12:23 AM on July 4, 2022 [18 favorites]
If you're interested in more detail and history, check out The Weather Machine by Andrew Blum!
posted by sriracha at 6:08 AM on July 4, 2022
posted by sriracha at 6:08 AM on July 4, 2022
Thanks for bringing this here.
My sweetheart is a sizeable part of making sure our Big Weather Eye in the Sky doesn't go blind in a few decades. It's how we pay the bills in this household. It's good to see awareness of some of the ways these government programs make everyone's lives better.
posted by tigrrrlily at 8:49 AM on July 4, 2022
My sweetheart is a sizeable part of making sure our Big Weather Eye in the Sky doesn't go blind in a few decades. It's how we pay the bills in this household. It's good to see awareness of some of the ways these government programs make everyone's lives better.
posted by tigrrrlily at 8:49 AM on July 4, 2022
Windy (where you can pick from ECMWF, GFS, ICON-EU, ICON, NEMS, and AROME)
Windy.com may be one of my favorite websites of all time. Just watch those winds blow.
posted by UN at 12:08 PM on July 4, 2022 [1 favorite]
Windy.com may be one of my favorite websites of all time. Just watch those winds blow.
posted by UN at 12:08 PM on July 4, 2022 [1 favorite]
One weather model researcher's opinion twenty years ago was that there wasn't much headroom left in model improvements, the need was for more measurements of current state (esp. over oceans), for the problem of "best estimate of conditions at time/place". People want to know whether the cloud will drop rain here or there; this fundamentally can't be predicted very far out no matter how smart the model, but knowing what's upwind in detail is critical. (Modeling local terrain features in enough resolution was also a gap, he said, but he left that to the hardware people to provide more compute power.)
Here's a few links to the US WPC verification data over the time since then.
This one for temperature is a good example; the average absolute error (deg F) for a 5-day forecast high is now about 4 degrees. 20 years ago, that 5-day error was more like 5.5 degrees. Now, even the 7th forecast day has an average absolute error under 5.5 degrees. A 7-day forecast high is more accurate today than a 5-day forecast was twenty years ago, and more accurate than a 3-day forecast was in the late 80s.
For rain, the measure is the 'threat score' which goes from 0 if none of the rain fell where it was predicted to 1 where all of the rain fell where it was predicted. A storm where South Dakota and Nebraska were predicted to get rain, but instead North and South Dakota did would get a score of 0.333 (assuming all three states are the same size). Note that this is based on a specific amount of rain; if the prediction is for half an inch of rain and Nebraska only got 3/8 of an inch, it's still considered wrong.
Anyways, here's the threat score for precipitation - 3-day rain forecasts in the early 2000s were about as good as 2-day forecasts in the early 90s and 1-day forecasts in the 1960s. Now 3-day rain forecasts are as good as 1-day forecasts were in the late 1990s. Longer range, twenty years ago, the forecast for days 4 and 5 were as good as early-70s 1-day forecasts, and now they're as good as late-90s 1-day forecasts. 6 and 7 day forecasts haven't gotten much better, but they're a little better today than 1-day forecasts in the 1960s.
posted by Superilla at 5:20 PM on July 4, 2022 [7 favorites]
Here's a few links to the US WPC verification data over the time since then.
This one for temperature is a good example; the average absolute error (deg F) for a 5-day forecast high is now about 4 degrees. 20 years ago, that 5-day error was more like 5.5 degrees. Now, even the 7th forecast day has an average absolute error under 5.5 degrees. A 7-day forecast high is more accurate today than a 5-day forecast was twenty years ago, and more accurate than a 3-day forecast was in the late 80s.
For rain, the measure is the 'threat score' which goes from 0 if none of the rain fell where it was predicted to 1 where all of the rain fell where it was predicted. A storm where South Dakota and Nebraska were predicted to get rain, but instead North and South Dakota did would get a score of 0.333 (assuming all three states are the same size). Note that this is based on a specific amount of rain; if the prediction is for half an inch of rain and Nebraska only got 3/8 of an inch, it's still considered wrong.
Anyways, here's the threat score for precipitation - 3-day rain forecasts in the early 2000s were about as good as 2-day forecasts in the early 90s and 1-day forecasts in the 1960s. Now 3-day rain forecasts are as good as 1-day forecasts were in the late 1990s. Longer range, twenty years ago, the forecast for days 4 and 5 were as good as early-70s 1-day forecasts, and now they're as good as late-90s 1-day forecasts. 6 and 7 day forecasts haven't gotten much better, but they're a little better today than 1-day forecasts in the 1960s.
posted by Superilla at 5:20 PM on July 4, 2022 [7 favorites]
Neat, kersplunk!
How many tails are there, like, how many axes of variation are usually considered?
posted by porpoise at 7:49 PM on July 4, 2022
How many tails are there, like, how many axes of variation are usually considered?
posted by porpoise at 7:49 PM on July 4, 2022
Hi porpoise, in stats the tail is just the extreme end of a distribution, so you can define it however you want. For example, if you had a model that simulated say, annual rainfall for Houston, you could run it 10,000 times and sort its output. That means that you can look at scenario 9,900 and predict that Houston will only have annual rainfall exceeding that once every 100 years.
The fatness of the tail is very important because it's a way of expressing uncertainty. How many people will crash their cars in the US this year? Probably about the same as last year. Will tropical storm X rapidly intensify into a hurricane and hit Miami three days later? A lot messier.
posted by kersplunk at 10:39 PM on July 4, 2022 [2 favorites]
The fatness of the tail is very important because it's a way of expressing uncertainty. How many people will crash their cars in the US this year? Probably about the same as last year. Will tropical storm X rapidly intensify into a hurricane and hit Miami three days later? A lot messier.
posted by kersplunk at 10:39 PM on July 4, 2022 [2 favorites]
How many tails are there, like, how many axes of variation are usually considered?
There are two tails, one of the low end and one of the high end, assuming a normal distribution (which you can google to see the values for). If you think of a statistic like adult height, then the tails will be people shorter than 4ft and taller than 7ft.
The fatness of the tail is very important because it's a way of expressing uncertainty.
Right and the uncertainty grows the more specific the prediction. How many cars will crash at the intersection of 3rd and Main in Portland Maine? Lots of uncertainty. How many named hurricanes will there be? Probably about the same number as last year.
posted by The_Vegetables at 7:20 AM on July 5, 2022
There are two tails, one of the low end and one of the high end, assuming a normal distribution (which you can google to see the values for). If you think of a statistic like adult height, then the tails will be people shorter than 4ft and taller than 7ft.
The fatness of the tail is very important because it's a way of expressing uncertainty.
Right and the uncertainty grows the more specific the prediction. How many cars will crash at the intersection of 3rd and Main in Portland Maine? Lots of uncertainty. How many named hurricanes will there be? Probably about the same number as last year.
posted by The_Vegetables at 7:20 AM on July 5, 2022
« Older California takes bold step to reduce truck... | Have Fun Newer »
This thread has been archived and is closed to new comments
posted by Nancy Lebovitz at 6:45 AM on July 3, 2022 [2 favorites]