THE PRAGMATIC PROFILE OF CHATGPT: Assessing the communicative skills of a conversational agent

This study examines the pragmatic abilities of OpenAI’s ChatGPT, a conversational agent based on the multi-layer Transformer network GPT-3.5. To do so, we administered a language battery to assess expressive and receptive pragmatic skills and compared the results with human performance. ChatGPT results were mostly human-like but revealed weaknesses in the domains of the Gricean maxim of quantity, text-based inferences, physical metaphors, and humor comprehension. On the one hand, these findings suggest that at least part of the linguistic pragmatic competence, as evaluated via current assessment tools, might be distributionally encoded in language. On the other hand, situated and meta-representational aspects of pragmatic inferencing appear to be not yet fully accounted for in LLMs.