Assessing the inter-rater agreement between observers, in the case of ordinal variables, is an important issue in both the statistical theory and biomedical applications. Typically, this problem has been dealt with the use of Cohen's weighted kappa, which is a modification of the original kappa statistic, proposed for nominal variables in the case of two observers. Fleiss (1971) put forth a generalization of kappa in the case of multiple observers, but both Cohen's and Fleiss' kappa could have a paradoxical behavior, which may lead to a difficult interpretation of their magnitude. In this paper, a modification of Fleiss' kappa, not affected by paradoxes, is proposed, and subsequently generalized to the case of ordinal variables. Monte Carlo simulations are used both to testing statistical hypotheses and to calculating percentile and bootstrap-t confidence intervals based on this statistic. The normal asymptotic distribution of the proposed statistic is demonstrated. Our results are applied to the classical Holmquist et al.'s (1967) dataset on the classification, by multiple observers, of carcinoma in situ of the uterine cervix. Finally, we generalize the use of s∗ to a bivariate case.

Marasini, D., Quatto, P., Ripamonti, E. (2016). Assessing the inter-rater agreement for ordinal data through weighted indexes. STATISTICAL METHODS IN MEDICAL RESEARCH, 25(6), 2611-2633 [10.1177/0962280214529560].

Assessing the inter-rater agreement for ordinal data through weighted indexes

MARASINI, DONATA;QUATTO, PIERO;RIPAMONTI, ENRICO
2016

Abstract

Assessing the inter-rater agreement between observers, in the case of ordinal variables, is an important issue in both the statistical theory and biomedical applications. Typically, this problem has been dealt with the use of Cohen's weighted kappa, which is a modification of the original kappa statistic, proposed for nominal variables in the case of two observers. Fleiss (1971) put forth a generalization of kappa in the case of multiple observers, but both Cohen's and Fleiss' kappa could have a paradoxical behavior, which may lead to a difficult interpretation of their magnitude. In this paper, a modification of Fleiss' kappa, not affected by paradoxes, is proposed, and subsequently generalized to the case of ordinal variables. Monte Carlo simulations are used both to testing statistical hypotheses and to calculating percentile and bootstrap-t confidence intervals based on this statistic. The normal asymptotic distribution of the proposed statistic is demonstrated. Our results are applied to the classical Holmquist et al.'s (1967) dataset on the classification, by multiple observers, of carcinoma in situ of the uterine cervix. Finally, we generalize the use of s∗ to a bivariate case.
Articolo in rivista - Articolo scientifico
Fleiss' kappa; inter-rater agreement; multiple observers; ordinal variables; weighted indexes;
Inter-rater agreement, Fleiss’ kappa, multiple observers, ordinal variables, weighted indexes
English
2016
25
6
2611
2633
none
Marasini, D., Quatto, P., Ripamonti, E. (2016). Assessing the inter-rater agreement for ordinal data through weighted indexes. STATISTICAL METHODS IN MEDICAL RESEARCH, 25(6), 2611-2633 [10.1177/0962280214529560].
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/51230
Citazioni
  • Scopus 66
  • ???jsp.display-item.citation.isi??? 57
Social impact