VERIFYING A 3 BY 3 CONFUSION MATRIX FORECAST
Written by: Indra Rahmawan and Halmar Halide
The verification of the three-category rainfall classification system in this study aims to ensure the accuracy of evaluation metric calculations, particularly the Peirce Skill Score (PSS), which is designed to assess the model’s ability to distinguish between events and non-events more effectively than random guessing. In addition, validation of the evaluation formula implementations was conducted by comparing the results against values published in previous studies, in order to assess the consistency and reliability of the methods used. This verification was applied to two main case studies: the study by Grazzini et al. (2024), which employed a machine learning approach to classify the physical mechanisms of extreme precipitation in Northern Italy, and the study by Benevides et al. (2019), which utilized artificial neural networks to predict hourly rainfall intensity based on GNSS and meteorological data.
The verification results for the Grazzini et al. study show that the developed evaluation program successfully replicated the evaluation metrics with a high level of agreement. The overall PSS value of 0.9072 closely matches the results reported in the journal, with strong performance across all categories: the orographic category (Cat1) achieved a PSS of 0.938 and other values such as the Threat Score (TS), Hit Rate (H), and False Alarm Ratio (F) indicated near-perfect accuracy; the frontal category (Cat2) recorded a PSS of 0.898, still reflecting very good performance despite being slightly lower due to overlapping characteristics with the convective category; and the convective category (Cat3) achieved a PSS of 0.866 with a very high θ value, indicating a strong correlation between prediction and observation. These findings reinforce the reliability of the multicategory verification approach using a 3×3 confusion matrix and the PSS metric as tools for evaluating the classification performance of machine learning-based weather forecasting systems. These results affirm that verification using multicategory PSS is highly effective in assessing the reliability of weather classification models based on machine learning. The verification program developed was proven capable of consistently reproducing metric accuracy in line with the referenced publication (Grazzini et al., 2024).
Meanwhile, the verification of the Benevides et al. (2019) study revealed differences in classification performance across categories. For the No Rain category, although TS and H values were very high and the False Alarm Ratio was very low, the PSS only reached 0.406. This indicates that the model frequently misclassified rainy events as dry, largely due to the dominance of No Rain data in the dataset. The Moderate Rain category showed the weakest performance, with a TS of only 0.319 and a PSS of 0.349, along with a low hit rate, reflecting the model’s difficulty in capturing moderate-intensity rainfall, which is physically more difficult to distinguish. In contrast, the Intense Rain category demonstrated better performance, with a PSS of 0.635 and a TS of 0.538, although notable false alarms were still present. The bias value for this category also indicated a slight tendency for the model to underpredict intense rainfall, although still within reasonable limits. Overall, the average PSS value of 0.3895 indicates moderate classification performance, with strengths in the extreme categories and a clear weakness in detecting moderate rain. The verification results reinforce the original findings of Benevides et al. (2019) that the model is more accurate in detecting extreme conditions (intense rain and no rain), but has a fundamental limitation in capturing the moderate rain category. The comparison between the two studies demonstrates that the verification program implemented in this research has successfully evaluated the performance of multicategory classification in an accurate and consistent manner.
[1] Grazzini, F., Craig, G. C., Keil, C., Antolini, G., & Pavan, V. (2020). Extreme precipitation events over northern Italy. Part I: A systematic classification with machine-learning techniques. Quarterly Journal of the Royal Meteorological Society, 146(726), 69–85. https://doi.org/10.1002/qj.3635
[2] Benevides, P., Catalao, J., & Nico, G. (2019). Neural network approach to forecast hourly intense rainfall using GNSS precipitable water vapor and meteorological sensors. Remote Sensing, 11(8). https://doi.org/10.3390/rs11080966
