Quality over quantity: focusing on high-conflict trials to improve the reliability and validity of attentional control measures

Conflict tasks such as the Stroop and the Simon are commonly employed to measure attentional control. Recent studies however, have consistently failed to find meaningful correlations between performance in these paradigms. While such findings may question the construct validity of attentional control measures, some authors have argued that the observed low correlations may also result from measurement error, as congruency effects typically display poor reliability in conflict tasks. In the present study we thus sought to improve the psychometric properties of the congruency effect in a spatial Stroop and a Simon task (N = 195), with the idea that more reliable attentional control measures should correlate across paradigms. Basing on well-replicated findings from experimental research, we argue that response conflict, and thus the need for attentional control, is weak in slow responses and following incongruent trials. We thus predicted that excluding these trials from analyses should improve the psychometric properties of the congruency effect. After controlling for these factors, not only we found excellent split-half reliability for the congruency effect in both paradigms, but we could also show robust between-task correlations.