Prevalence dependence in model goodness measures with special emphasis on true skill statistics

• Published in: Ecology and evolution Vol. 7; no. 3; pp. 863 - 872
• Main Authors: Somodi, Imelda, Lepesi, Nikolett, Botta‐Dukát, Zoltán
• Format: Journal Article
• Language: English
• Published: England John Wiley & Sons, Inc 01.02.2017; John Wiley and Sons Inc
• Subjects: Cohen's kappa; Data structures; Datasets; Dependence; Discrimination; Ecological monitoring; Introduced species; Kappa coefficient; model performance; Original Research; predictive models; sample size; species distribution models; Statistical methods; Statistics; predictive models; sample size; species distribution models; Cohen's kappa; model performance
• ISSN: 2045-7758; 2045-7758
• DOI: 10.1002/ece3.2654

It has long been a concern that performance measures of species distribution models react to attributes of the modeled entity arising from the input data structure rather than to model performance. Thus, the study of Allouche et al. (Journal of Applied Ecology, 43, 1223, 2006) identifying the true skill statistics (TSS) as being independent of prevalence had a great impact. However, empirical experience questioned the validity of the statement. We searched for technical reasons behind these observations. We explored possible sources of prevalence dependence in TSS including sampling constraints and species characteristics, which influence the calculation of TSS. We also examined whether the widespread solution of using the maximum of TSS for comparison among species introduces a prevalence effect. We found that the design of Allouche et al. (Journal of Applied Ecology, 43, 1223, 2006) was flawed, but TSS is indeed independent of prevalence if model predictions are binary and under the strict set of assumptions methodological studies usually apply. However, if we take realistic sources of prevalence dependence, effects appear even in binary calculations. Furthermore, in the widespread approach of using maximum TSS for continuous predictions, the use of the maximum alone induces prevalence dependence for small, but realistic samples. Thus, prevalence differences need to be taken into account when model comparisons are carried out based on discrimination capacity. The sources we identified can serve as a checklist to safely control comparisons, so that true discrimination capacity is compared as opposed to artefacts arising from data structure, species characteristics, or the calculation of the comparison measure (here TSS). It has long been a concern that performance measures of species distribution models (SDM) react to attributes of the modeled entity arising from the input data structure (including the ratio of presences and absences known as prevalence) rather than to model performance. The true skill statistics (TSS) has been propagated as unaffected by prevalence changes; however, experience questioned this. Therefore, we examined possible causes of observed prevalence dependence for TSS, while also extending the theory of prevalence dependence in general.