Investigating The Impact Of Touch On Multisensory Learning In Mixed Reality Environments

As a cornerstone of medical education cadaveric dissection maintained its position as a gold- standard for anatomy education throughout history. Research showed that it outperformed classical illustration-based alternatives and even some interactive alternatives like the dissection tables that lacked tactile feedback. Emerging mixed reality (MR) technologies are offering not only immersive but also embodiment features and are raising interest in anatomy education due to their identified equivalent learning performance to dissection. From a neuroscience perspective, haptics and sensorimotor learning rely on action-perception coupling and sensory motor synchrony as central elements shaping the degree of realism offered by MR. How realistic touch interferes with emotions and cognition is little studied in literature due to the complexity of assessing tactile perception separately from the motor component. Meta Quest 3 through the virtual hands embodied features is offering new possibilities to understand the role of the motor component of haptics separately from touch. The present study investigates how realistic augmented reality (AR) (motor and tactile), and non-realistic augmented virtuality (AV) (motor only) haptic interactions influence short-term memory, emotions, and learning anatomy in MR environments. A particular attention was given to the role of sex and VR familiarity in shaping these dynamics. 40 healthcare students participated each in one of the learning conditions AR or AV. They completed sensory short-term memory (STM) assessment, knowledge tests, emotional ratings, usability questionnaires, and a short interview on the cognitive strategies used to recall sensory information. Both conditions demonstrated good usability and produced significant improvement in learning gain (LG), visual STM, and tactile STM. Auditory STM did not improve. Intergroup technology-based, and sex-based differences were observed when assessing emotional co-occurrence patterns, and how emotions shape memory performance, the relevant cognitive strategies used and their interaction with LG. Such processes suggest sensorimotor learning is modulated by affect in MR environments, and that emotions modulate the allocation of executive resources toward certain sensory functions depending on context, technology, and sex. The study provides an ecological neuroscience-based framework for designing MR anatomy education experiences and suggests that spatial learning is a multidimensional process that relies on a network of multiples factors where emotions could play an activating or disactivating role in learning and could induce in isolation or as group synergistic or competitive effects in allocating attentional and executive resources towards specific sensory functions. These findings have strong implications in research on the role of affect in shaping haptic experiences with a translational value that encompasses education to interest neurorehabilitation using MR technologies.