The Fleiss’ kappa statistic is a well-known index for assessing the reliability of agreement between raters. It is used both in the psychological and in the psychiatric field. Unfortunately, the kappa statistic may behave inconsistently in case of strong agreement between raters, since this index assumes lower values than it would have been expected. The aim of this paper is to propose a new method to avoid this paradox through permutation techniques. Furthermore, we study the problem of kappa confidence intervals and, in particular, we suggest to use Bootstrap confidence intervals free of paradoxes.

Falotico, R., Quatto, P. (2015). Fleiss’ Kappa statistic without paradoxes. QUALITY & QUANTITY, 49(2), 463-470 [10.1007/s11135-014-0003-1].

Fleiss’ Kappa statistic without paradoxes

FALOTICO, ROSA
;
QUATTO, PIERO
2015

Abstract

The Fleiss’ kappa statistic is a well-known index for assessing the reliability of agreement between raters. It is used both in the psychological and in the psychiatric field. Unfortunately, the kappa statistic may behave inconsistently in case of strong agreement between raters, since this index assumes lower values than it would have been expected. The aim of this paper is to propose a new method to avoid this paradox through permutation techniques. Furthermore, we study the problem of kappa confidence intervals and, in particular, we suggest to use Bootstrap confidence intervals free of paradoxes.
Articolo in rivista - Articolo scientifico
Inter-rater agreement; Fleiss’ kappa; Kappa paradoxes; Monte Carlo simulations; Bootstrap confidence intervals
English
2015
49
2
463
470
reserved
Falotico, R., Quatto, P. (2015). Fleiss’ Kappa statistic without paradoxes. QUALITY & QUANTITY, 49(2), 463-470 [10.1007/s11135-014-0003-1].
File in questo prodotto:
File Dimensione Formato  
FleissKappaStatisticWithoutPar.pdf

Solo gestori archivio

Tipologia di allegato: Publisher’s Version (Version of Record, VoR)
Dimensione 133.97 kB
Formato Adobe PDF
133.97 kB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/51436
Citazioni
  • Scopus 113
  • ???jsp.display-item.citation.isi??? 93
Social impact