Insensitivity and Oversensitivity to Answer Diagnosticity in Hypothesis Testing

Two experiments examined how people perceive the diagnosticity of different answers (“yes” and “no”) to the same question. We manipulated whether the “yes” and the “no” answers conveyed the same amount of information or not, as well as the presentation format of the probabilities of the features inquired about. In Experiment 1, participants were presented with only the percentages of occurrence of the features, which most straightforwardly apply to the diagnosticity of “yes” answers. In Experiment 2, participants received in addition the percentages of the absence of features, which serve to assess the diagnosticity of “no” answers. Consistent with previous studies, we found that participants underestimated the difference in the diagnosticity conveyed by different answers to the same question. However, participants’ insensitivity was greater when the normative (Bayesian) diagnosticity of the “no” answer was higher than that of the “yes” answer. We also found oversensitivity to answer diagnosticity, whereby participants valued as differentially diagnostic two answers that were normatively equal in terms of their diagnosticity. Presenting to participants the percentages of occurrence of the features inquired about together with their complements increased their sensitivity to the diagnosticity of answers. We discuss the implications of these findings for confirmation bias in hypothesis testing.