Evaluation of inter-observer reliability in the case of trichotomous and four-level animal-based welfare indicators with two observers

This study focuses on assessing inter-observer reliability (IOR) between two observers in the case of trichotomous and four-level animal-based welfare indicators assessed at individual level. The Body Condition Score (BCS) and Knee calluses (KNC) were chosen as trichotomous indicators; data were collected in fourteen intensively managed dairy goat farms in Italy (ITF1 to ITF7) and Portugal (PTF1 to PTF7) and in extensively managed dairy goat farms exploiting three alpine pastures (AP1, AP2 and AP3) in Italy. The Ear posture (EP) and Eye white (EW) were chosen as four-level indicators; data were collected in three intensively managed dairy cattle farms (F1, F2 and F3) in Italy. The performance of the most documented agreement indices was compared. In the case of trichotomous indicators, Scott’s π, Cohen’s K, Cohen’s KC, Cohen’s weighted K and Krippendorff’s α were affected by the paradox effect: when the concordance rate (P0) was high, they sometimes gave very low or even negative values (e.g. P0(BCS-ITF3) = 74%; Scott’s π = 0.05; Cohen’s K = 0.09; Krippendorff’s α = 0.06; P0(BCS-AP3) = 74%; Scott’s π = −0.12; Cohen’s K = Krippendorff’s α = −0.11). Bangdiwala’s B, Gwet’s γ(AC1) and Quatto’s weighted S were not affected by this phenomenon and provided values very close to P0 (e.g. P0(KNC-PTF1) = 88%; Bangdiwala’s B = Gwet’s γ(AC1) = 0.85; P0(BCS-AP1) = 82%; Bangdiwala’s B = Gwet’s γ(AC1) = 0.79). In the case of four-level indicators, Cohen’s K and Krippendorff’s α were not affected by the paradox behaviour. However, Cohen’s KC in some cases exceeded the observed P0 (e.g. P0(EP-F3) = 78%; Cohen’s KC = 1). Gwet’s γ(AC1) showed the best results for four-level indicators (e.g. P0(EP-F1) = 88%; Gwet’s γ(AC1) = 0.86), followed by Quatto’s S and Holley and Guilford’s G (e.g. P0(EP-F1) = 88%; Quatto’s S = Holley and Guilford’s G = 0.84). To evaluate IOR between two observers, Bangdiwala’s B, Gwet’s γ(AC1) and Quatto’s weighted S are suggested for trichotomous indicators, while Gwet’s γ(AC1), Quatto’s S and Holley and Guilford’s G are suggested for four-level indicators.