Causal inference modulates audiovisual temporal recalibration

Poster Presentation 43.472: Monday, May 22, 2023, 8:30 am – 12:30 pm, Pavilion
Session: Multisensory Processing: Audio-visual, visuo-vestibular

Luhe Li1 (), Fangfang Hong1, Stephanie Badde2, Michael S. Landy1,3; 1Department of Psychology, New York University, 2Department of Psychology, Tufts University, 3Center for Neural Science, New York University

Cross-modal recalibration promotes perceptual accuracy by correcting for systematic discrepancies between multisensory cues. Recently, it has been shown that cross-modal spatial recalibration relies on observers’ multisensory percepts, which in turn are guided by causal inference. Causal inference solves the central challenge for human observers to integrate cues from multiple modalities that derive from the same source, but segregate cues from separate sources. Yet, it remains unknown whether this framework generalizes to the temporal domain. We adopted a classical recalibration paradigm composed of a pre-test, an exposure phase, and a post-test repeated across nine sessions. In pre- and post-tests, we measured observers’ perceived audiovisual synchrony: observers indicated the perceived order (“vision first,” “audition first,” or “simultaneous”) of audiovisual stimulus pairs with varying temporal offset. In the exposure phase, observers were exposed in each session to 250 audiovisual stimuli with one of nine fixed temporal offsets that differed in sign and magnitude. In the subsequent post-test, perceived synchrony was shifted in the direction of the temporal offset experienced during the exposure phase. Moreover, the amount of recalibration first increased but then decreased with increasing temporal offset (audio lead or lag) during the exposure phase. This pattern agrees with an involvement of causal inference because temporally distant stimuli are more likely to originate from separate causes and therefore their perceptual estimates should lead to less recalibration. To further test our hypothesis, we fitted several variants of the causal-inference model to the temporal-order judgments as well as a model that assumes recalibration by a constant proportion of the adaptor temporal offset. Model comparison revealed that the majority of observers relied on causal-inference-driven percepts during recalibration, indicating that causal inference plays a key role for multisensory perception in the temporal domain.

Acknowledgements: This work was supported by grant NIH EY08266.