Tableau maps
To visually depict the accuracy of the LSTM, our best-performing model, we plotted multiple Tableau heat maps to highlight county-level trends among the actual versus predicted drought scores and their corresponding evaluation metrics. Please see below for more information about what each page includes, how to read the maps, and the key takeways.
We've created two pages of county-level maps, accessible via the below tabs:
drought scoring
Showcases actual drought scores for the last 12 weeks in the dataset and corresponding score predictions for those 1-12 weeks using the previous 30 weeks, the aforementioned standard prediction timeframe, of past data
By clicking on the right arrow "play" button below, you can observe the changes in drought score over time for both maps
Hovering your cursor over the maps will bring up a Tooltip with the county name, week number, and the predicted/actual drought score
evaluation metrics
Showcases the calculated mean absolute error (MAE), mean squared error (MSE), and macro F1 scores using a 30-week window and a 12-week horizon
The lower the MSE and MAE and the higher the F1 score, then the better the accuracy
Hovering your cursor over the maps will bring up a Tooltip with the county name, MAE/MSE/macro F1 values, average precipitation and average Earth skin temperature
Note: it may be difficult to interact with the below Tableau maps on the mobile version of the website.
Interpreting the Maps
drought scoring
For this particular evaluation window, the LSTM performed best in counties that experience more significant and consistent drought conditions. For example, the LSTM produced an average score discrepancy of as low as 0.12 drought categories in Lake and Yolo counties, which had average drought scores around 4 over the 12 week period
The model predictions were very accurate for the first 7 weeks, deviating less than 0.1 in score; beginning in week 8, the difference between predicted and actual average scores grew substantially, particularly in weeks 10-12
It appeared the greatest drought score discrepancies lay in counties along the Nevada border leading up to northern California; however, we found the greatest differences were due to rapid increase in drought scores over 2-3 weeks
For weeks 10-12, the maps illustrated that in San Luis Obispo, Santa Barbara, and Ventura counties, the model underpredicted by two drought categories
These counties had actual drought scores of around 2, while the model predicted no drought conditions, or a score of 0
As the first seven weeks in these counties had no drought, the model predicted a continuation of no drought conditions
Other counties that experienced lower model accuracies were Los Angeles and Monterey, which both had average score discrepancies of ~1.9 drought categories
Unlike in San Luis Obispo, Santa Barbara, and Ventura counties, these two all had actual and predicted drought scores greater than 0; however, the differences here could be attributed to the actual drought scores being lower than 1 for the first 8-9 weeks, before jumping to 2 in the final 3-4 weeks of our horizon
Therefore, the model only predicted drought scores of less than 0.5 for Los Angeles and Monterey due to most of the timeframe experiencing insignificant drought conditions
evaluation metrics
Maps illustrating MAE and MSE depict the disparity between predicted and actual drought scores, revealing consistent trends. Specifically, the models demonstrated lower prediction errors in the central region of the state
F1 scores exceeded 0.86 for more than half of the counties assessed
Certain areas in the southeast of the state, notably counties along the Nevada-California border, exhibited low F1 scores
The F1 score exhibited a strong correlation with the severe drought ratio, calculated as the number of severe drought cases divided by all samples in the test data, within each county (correlation coefficient = 0.68)
When severe drought ratio fell below 0.7%, notably low F1 scores (~0.5) were observed, indicating the model's inability to predict severe drought occurrences in these areas
F1 scores below 0.65 were commonly associated with severe drought ratios below 4% in the respective counties, suggesting a tendency for the models to underestimate severe drought occurrences