Multimodal Prior Knowledge Determines False Memory Formation

Memory formation is a complex phenomenon shaped by various experiential traces, yet their exact contributions remain unclear. This study investigates the generation of false memories leveraging different data-driven computational models to independently quantify language-based and vision-based experiential knowledge, as extracted from large-scale databases consisting of 639 billion words and 15 million images, respectively. We then tested the effects of these knowledge sources in two false memory experiments, one employing images and the other words as stimuli. Our findings unveil both modality-independent and modality-dependent processes in the formation of memory traces. Indeed, we observed a contribution of both prior visual and linguistic knowledge regardless of the types of stimuli to be memorized. However, the extent of this contribution differed as a function of the modality tested: Visual prior knowledge is more influential in image-based tasks, while linguistic prior knowledge dominates in word-based tasks. This dual and modality-dependent contribution underscores the adaptive nature of memory processes, revealing the dynamic integration of diverse experiential traces in false memory formation.