Implicit Measures as a Useful Tool for Evaluating User Experience

With the increasing presence in our lives of interactive and technological items, the concept of “experience” becomes crucial in both the design and the evaluation phases. When aiming to measure such experience, researchers typically adopt self-report methodology. However, it has been established that also unconscious motives (i.e., emotions, implicit attitudes) - which people are not always aware of - need to be taken into account in understanding users’ perception, by means of indirect measures. The present study investigates the possibility of measuring UX by both implicit and explicit attitudes measures. In an experimental study (N = 36), we asked participants to evaluate a conversational chatbot prototype, which had the fictional goal of pre-selecting candidates for a job, and to fix a date for the job interview. We manipulated between-participants the chatbot’s gender (male vs. female) and the tone of voice (formal vs. informal). Explicit user experience’s evaluation (i.e., UEQ) and implicit attitudes towards the chatbot (i.e., IAT) were the dependent variables, as well as self-reported measures of chatbot’s efficacy and emotional experience. Our findings showed an implicit preference towards the informal version of the chatbot, as revealed by the IAT scores, whereas no differences emerged on explicit measures of the UEQ. Results are discussed in light of the effect of implicit attitudes on overall experience, and on the importance of comparing implicit and explicit measures, as well as objective usability measures, when evaluating the UX.