The Fleiss’ kappa statistic is a well-known index for assessing the reliability of agreement between raters. It is used both in the psychological and in the psychiatric field. Unfortunately, the kappa statistic may behave inconsistently in case of strong agreement between raters, since this index assumes lower values than it would have been expected. The aim of this paper is to propose a new method to avoid this paradox through permutation techniques. Furthermore, we study the problem of kappa confidence intervals and, in particular, we suggest to use Bootstrap confidence intervals free of paradoxes.
Falotico, R., Quatto, P. (2015). Fleiss’ Kappa statistic without paradoxes. QUALITY & QUANTITY, 49(2), 463-470 [10.1007/s11135-014-0003-1].
Fleiss’ Kappa statistic without paradoxes
FALOTICO, ROSA
;QUATTO, PIERO
2015
Abstract
The Fleiss’ kappa statistic is a well-known index for assessing the reliability of agreement between raters. It is used both in the psychological and in the psychiatric field. Unfortunately, the kappa statistic may behave inconsistently in case of strong agreement between raters, since this index assumes lower values than it would have been expected. The aim of this paper is to propose a new method to avoid this paradox through permutation techniques. Furthermore, we study the problem of kappa confidence intervals and, in particular, we suggest to use Bootstrap confidence intervals free of paradoxes.
| Campo DC | Valore | Lingua |
|---|---|---|
| dc.authority.academicField2024 | Settore STAT-01/A - Statistica | * |
| dc.authority.ancejournal | QUALITY & QUANTITY | - |
| dc.authority.people | FALOTICO, ROSA | en |
| dc.authority.people | QUATTO, PIERO | en |
| dc.collection.id.s | e39773c1-7ce2-35a3-e053-3a05fe0aac26 | * |
| dc.collection.name | 01 - Articolo su rivista | * |
| dc.contributor.appartenenza | DIPARTIMENTO DI ECONOMIA, METODI QUANTITATIVI E STRATEGIE DI IMPRESA | * |
| dc.contributor.appartenenza.mi | 4406 | * |
| dc.contributor.area | AREA MIN. 13 - SCIENZE ECONOMICHE E STATISTICHE | * |
| dc.date.accessioned | 2014/05/06 14:14:54 | - |
| dc.date.available | 2014/05/06 14:14:54 | - |
| dc.date.issued | 2015 | en |
| dc.date.submission | 2019/02/18 16:43:34 | * |
| dc.description.abstracteng | The Fleiss’ kappa statistic is a well-known index for assessing the reliability of agreement between raters. It is used both in the psychological and in the psychiatric field. Unfortunately, the kappa statistic may behave inconsistently in case of strong agreement between raters, since this index assumes lower values than it would have been expected. The aim of this paper is to propose a new method to avoid this paradox through permutation techniques. Furthermore, we study the problem of kappa confidence intervals and, in particular, we suggest to use Bootstrap confidence intervals free of paradoxes. | - |
| dc.description.allpeople | Falotico, R; Quatto, P | - |
| dc.description.allpeopleoriginal | Falotico, R; Quatto, P | en |
| dc.description.fulltext | reserved | en |
| dc.description.fulltextoriginal | mixed | en |
| dc.description.international | No | en |
| dc.description.numberofauthors | 2 | - |
| dc.identifier.citation | Falotico, R., Quatto, P. (2015). Fleiss’ Kappa statistic without paradoxes. QUALITY & QUANTITY, 49(2), 463-470 [10.1007/s11135-014-0003-1]. | en |
| dc.identifier.doi | 10.1007/s11135-014-0003-1 | en |
| dc.identifier.isi | WOS:000352509200003 | - |
| dc.identifier.scopus | 2-s2.0-84957962535 | - |
| dc.identifier.uri | http://hdl.handle.net/10281/51436 | - |
| dc.language.iso | eng | en |
| dc.publisher.name | Springer Netherlands | - |
| dc.relation.firstpage | 463 | en |
| dc.relation.issue | 2 | en |
| dc.relation.lastpage | 470 | en |
| dc.relation.numberofpages | 8 | en |
| dc.relation.volume | 49 | en |
| dc.subject.keywords | Bootstrap confidence intervals; Fleiss’ kappa; Inter-rater agreement; Kappa paradoxes; Monte Carlo simulations; | - |
| dc.subject.keywordseng | Inter-rater agreement; Fleiss’ kappa; Kappa paradoxes; Monte Carlo simulations; Bootstrap confidence intervals | - |
| dc.subject.singlekeyword | Inter-rater agreement | * |
| dc.subject.singlekeyword | Fleiss’ kappa | * |
| dc.subject.singlekeyword | Kappa paradoxes | * |
| dc.subject.singlekeyword | Monte Carlo simulations | * |
| dc.subject.singlekeyword | Bootstrap confidence intervals | * |
| dc.title | Fleiss’ Kappa statistic without paradoxes | en |
| dc.type | Articolo su rivista | - |
| dc.type.circulation | Rilevanza internazionale | it |
| dc.type.contribution | Articolo in rivista - Articolo scientifico | en |
| dc.type.driver | info:eu-repo/semantics/article | - |
| dc.type.full | Pubblicazioni::01 - Articolo su rivista | it |
| dc.type.impactfactor | si | it |
| dc.type.miur | 262 | en |
| dc.type.publicationstatus | Pubblicato | en |
| dc.type.referee | Esperti anonimi | en |
| dc.type.research | Scientifica | en |
| iris.isi.extIssued | 2015 | - |
| iris.isi.extTitle | Fleiss' kappa statistic without paradoxes | - |
| iris.mediafilter.data | 2025/04/05 13:30:08 | * |
| iris.orcid.lastModifiedDate | 2023/06/02 15:38:20 | * |
| iris.orcid.lastModifiedMillisecond | 1685713100931 | * |
| iris.scopus.extIssued | 2015 | - |
| iris.scopus.extTitle | Fleiss’ kappa statistic without paradoxes | - |
| iris.sitodocente.maxattempts | 1 | - |
| iris.unpaywall.doi | 10.1007/s11135-014-0003-1 | * |
| iris.unpaywall.isoa | false | * |
| iris.unpaywall.journalisindoaj | false | * |
| iris.unpaywall.metadataCallLastModified | 03/04/2025 04:13:42 | - |
| iris.unpaywall.metadataCallLastModifiedMillisecond | 1743646422867 | - |
| iris.unpaywall.oastatus | closed | * |
| isi.authority.ancejournal | QUALITY & QUANTITY###0033-5177 | * |
| isi.category | XY | * |
| isi.category | WU | * |
| isi.contributor.affiliation | University of Milano-Bicocca | - |
| isi.contributor.affiliation | University of Milano-Bicocca | - |
| isi.contributor.country | Italy | - |
| isi.contributor.country | Italy | - |
| isi.contributor.name | Rosa | - |
| isi.contributor.name | Piero | - |
| isi.contributor.researcherId | EWO-9499-2022 | - |
| isi.contributor.researcherId | GVS-8344-2022 | - |
| isi.contributor.subaffiliation | Dept Econ Management & Stat | - |
| isi.contributor.subaffiliation | Dept Econ Management & Stat | - |
| isi.contributor.surname | Falotico | - |
| isi.contributor.surname | Quatto | - |
| isi.date.issued | 2015 | * |
| isi.description.allpeopleoriginal | Falotico, R; Quatto, P; | * |
| isi.document.sourcetype | WOS.SCI | * |
| isi.document.type | Article | * |
| isi.document.typenorm | Articolo in rivista - Articolo scientifico | * |
| isi.document.types | Article | * |
| isi.identifier.doi | 10.1007/s11135-014-0003-1 | * |
| isi.identifier.eissn | 1573-7845 | * |
| isi.identifier.isi | WOS:000352509200003 | * |
| isi.journal.journaltitle | QUALITY & QUANTITY | * |
| isi.journal.journaltitleabbrev | QUAL QUANT | * |
| isi.language.original | English | * |
| isi.publisher.place | VAN GODEWIJCKSTRAAT 30, 3311 GZ DORDRECHT, NETHERLANDS | * |
| isi.relation.firstpage | 463 | * |
| isi.relation.issue | 2 | * |
| isi.relation.lastpage | 470 | * |
| isi.relation.volume | 49 | * |
| isi.title | Fleiss' kappa statistic without paradoxes | * |
| scopus.authority.ancejournal | QUALITY & QUANTITY###0033-5177 | * |
| scopus.category | 2613 | * |
| scopus.category | 3300 | * |
| scopus.contributor.affiliation | University of Milan-Bicocca | - |
| scopus.contributor.affiliation | University of Milan-Bicocca | - |
| scopus.contributor.afid | 60012306 | - |
| scopus.contributor.afid | 60012306 | - |
| scopus.contributor.auid | 56029633300 | - |
| scopus.contributor.auid | 6503868442 | - |
| scopus.contributor.country | Italy | - |
| scopus.contributor.country | Italy | - |
| scopus.contributor.dptid | 113551627 | - |
| scopus.contributor.dptid | 113551627 | - |
| scopus.contributor.name | Rosa | - |
| scopus.contributor.name | Piero | - |
| scopus.contributor.subaffiliation | Department of Economics;Management and Statistics; | - |
| scopus.contributor.subaffiliation | Department of Economics;Management and Statistics; | - |
| scopus.contributor.surname | Falotico | - |
| scopus.contributor.surname | Quatto | - |
| scopus.date.issued | 2015 | * |
| scopus.description.abstracteng | The Fleiss’ kappa statistic is a well-known index for assessing the reliability of agreement between raters. It is used both in the psychological and in the psychiatric field. Unfortunately, the kappa statistic may behave inconsistently in case of strong agreement between raters, since this index assumes lower values than it would have been expected. The aim of this paper is to propose a new method to avoid this paradox through permutation techniques. Furthermore, we study the problem of kappa confidence intervals and, in particular, we suggest to use Bootstrap confidence intervals free of paradoxes. | * |
| scopus.description.allpeopleoriginal | Falotico R.; Quatto P. | * |
| scopus.differences | scopus.description.allpeopleoriginal | - |
| scopus.document.type | ar | * |
| scopus.document.types | ar | * |
| scopus.identifier.doi | 10.1007/s11135-014-0003-1 | * |
| scopus.identifier.eissn | 1573-7845 | * |
| scopus.identifier.pui | 608240975 | * |
| scopus.identifier.scopus | 2-s2.0-84957962535 | * |
| scopus.journal.sourceid | 19372 | * |
| scopus.language.iso | eng | * |
| scopus.publisher.name | Springer Netherlands | * |
| scopus.relation.firstpage | 463 | * |
| scopus.relation.issue | 2 | * |
| scopus.relation.lastpage | 470 | * |
| scopus.relation.volume | 49 | * |
| scopus.subject.keywords | Bootstrap confidence intervals; Fleiss’ kappa; Inter-rater agreement; Kappa paradoxes; Monte Carlo simulations; | * |
| scopus.title | Fleiss’ kappa statistic without paradoxes | * |
| scopus.titleeng | Fleiss’ kappa statistic without paradoxes | * |
| Appare nelle tipologie: | 01 - Articolo su rivista | |
File in questo prodotto:
| File | Dimensione | Formato | |
|---|---|---|---|
|
FleissKappaStatisticWithoutPar.pdf
Solo gestori archivio
Tipologia di allegato:
Publisher’s Version (Version of Record, VoR)
Dimensione
133.97 kB
Formato
Adobe PDF
|
133.97 kB | Adobe PDF | Visualizza/Apri Richiedi una copia |
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
