Report-back: How well do satellite "data products"
estimate rain vs. snow? 

When it comes to figuring out what is falling from the sky using technology, scientists have multiple tools available, but none are perfect. It is important to understand how reliable each of these tools is so that we can learn more about the data they provide. We can also start looking for ways to improve them, which would ultimately help scientists with crucial tasks like estimating snowfall and rainfall.

With the help of your observations, Mountain Rain or Snow has taken a closer look at several of these tools in the past (you can read more about those studies here), but there are others that have not yet been comprehensively assessed – meaning we don’t have a great idea of how well they do at making accurate predictions of precipitation type. To address this, we recently took a closer look at three different satellite-based tools – IMERG, MERRA-2, and NLDAS-2 – to improve our understanding of these technologies’ performances for predicting what is falling from the sky. You can read the manuscript published by Guo Yu, PhD and our colleagues here.

IMERG, MERRA-2, and NLDAS-2 are all "data products", or large datasets available to researchers, that make estimates of precipitation type based on satellite observation. Each of these data products have their own method of putting information together to reach a best guess of what is falling from the sky. You can imagine these data products like three different detectives working independently on the same case – they may each find different clues and may even reach different conclusions, but they share the same goal of solving the mystery, and ultimately, there is only one correct answer. Like a firsthand witness, the real-time observations submitted to Mountain Rain or Snow unveil the mystery for thousands of cases, allowing us to assess how often each of these data product detectives correctly solve the case.

To evaluate how well these three data products perform at predicting precipitation type, we compared 37,338 observations from Mountain Rain or Snow between January 2020 to July 2023 to the predictions made by IMERG, NLDAS-2, and MERRA-2 for the same times and locations. Then, we calculated success metrics for each of the data products, taking several perspectives on their performances. For a map of how these observations were distributed across the United States, check out Figure 1.

Overall, we found that NLDAS-2 tended to have a bias towards rain, which meant it had a tendency to predict rain even when it was actually snowing. Although IMERG and MERRA-2 were less biased than NLDAS-2, all three data products struggled to predict snow above freezing as the temperature rose and struggled to predict rain below freezing as the temperature fell. This means phenomena like sub-freezing rain (rain that falls when the temperature outside is below 0°C) can go unnoticed by these data products.

We can see this performance trend for all three data products in Figure 2, which shows the Critical Success Index (CSI) of each data product for rain and snow. The CSI is a way of measuring each data product’s performance for rain and snow individually by comparing the number of cases when the data product made the correct prediction for the precipitation type in question to all its total predictions except the ones that were correct predictions of the other type. If you’re curious, check out the “Diving into Critical Success Index” at the bottom of the page for a walkthrough of the calculation. A value of 1.00 on the y-axis means the data product estimated perfectly every time, and smaller values (lighter colors on Figure 2) mean it did a progressively worse job of “solving the case”

How well the data products performed also showed some regional trends. All three data products tended to have higher CSI values – meaning they did a better job – in low elevation regions compared to higher elevation regions. Also, there were some regions where both IMERG and MERRA-2 seemed to struggle with certain precipitation types. For example, in the Northern Rockies, it seemed that both data products might have tended to predict snow when it was actually raining.

We’ve learned a lot about these data products, but we need more information to fill remaining gaps in our understanding. For example, since most Mountain Rain or Snow observations are submitted in the winter, we can’t say for certain that these results describe the data products’ performances comprehensively across seasons. The project also tends to receive far more observations of snow than rain and mixed precipitation (for this dataset, 61% snow compared to 27% rain and 12% mix), which may have affected the overall picture of the data products’ performances. Geographic factors might play a role, too; Mountain Rain or Snow observations tend to be clustered at some elevations more than others. Figure 3 shows the distribution of elevations.

With these factors in mind, to learn more about research questions like this in the future it is especially helpful to receive observations for a variety of situations and locations. Can you think of ways to help make the dataset more comprehensive? For an idea of elevational gaps in your area, check out this report-back about elevational diversity. With your help filling the gaps, we can continue to refine our understanding of these data product detectives’ ability to correctly answer the impactful question, “What is falling from the sky?”

Yu, G., Jennings, K.S., Hatchett, B.J., Nolin, A.W., Hur, N., Collins, M., Heggli, A., Tonino, S., & Arienzo, M.M. (2024). Crowdsourced data reveal shortcomings in precipitation phase products for rain and snow partitioning. Geophysical Research Letters, 51(24). https://doi.org/10.1029/2024GL112853.

Critical Success Index (CSI) is calculated individually for rain and snow. Here are the steps:

1. Add up all the cases when the data product made a correct prediction of the precipitation type we’re looking at (just rain or just snow) plus all the cases when the data product failed to make the correct prediction (for either rain or snow).

2. Divide the number of cases when the data product correctly predicted the precipitation type we’re looking at (just rain or just snow) by the number found in step 1.

For example, here is the calculation of CSI for rain:

So, the only situation not considered in calculating the CSI for rain is the situation that the data product correctly predicted snow. Similarly, the CSI for snow considers every situation when snow either was predicted or should have been predicted but doesn’t consider the situation that rain was correctly predicted.

The CSI values are calculated individually for each data product, so in total there are six unique CSI values: CSI for rain for IMERG, CSI for rain for MERRA-2, CSI for rain for NLDAS-2, CSI for snow for IMERG, CSI for snow for MERRA-2, and CSI for snow for NLDAS-2.