Visual representations shift from a retinal to a real-world reference frame during visual working memory

Poster Presentation 43.430: Monday, May 22, 2023, 8:30 am – 12:30 pm, Pavilion
Session: Visual Working Memory: Neural mechanisms

Maria V. Servetnik1,4 (), Nicolas Pollán Hauer2, Michael J. Wolff1, Chaipat Chunharas3, Rosanne L. Rademaker1; 1Ernst Strüngmann Institute (ESI) for Neuroscience in Cooperation with the Max Planck Society, Frankfurt, Germany, 2Centre de Recerca Matemàtica, Campus de Bellaterra, Barcelona, Spain, 3Department of Medicine, King Chulalongkorn Memorial Hospital, Chulalongkorn University, Bangkok, Thailand, 4Department of Cognitive Neuroscience, Vrije Universiteit Amsterdam, The Netherlands

Visual information in our everyday environment is anchored to a real-world reference frame – a tall building remains upright, even when you tilt your head, which changes the projection of the building on your retina from a vertical to a diagonal orientation. Presumably, retinotopic cortex represents perceptual inputs in a retinal reference frame. But what about visual information held in working memory? Here, we investigate which reference frame the brain uses during visual working memory by dissociating retinal and real-world reference frames via head tilt, combined with 64-channel electroencephalography. Nineteen participants completed between 2592–3168 trials during which they briefly viewed (150ms) and then remembered (1500ms) a randomly oriented grating for recall via method-of-adjustment. In interleaved blocks of trials, the participant's head was either kept upright, or tilted by 45º using a custom rotating chinrest. The remembered orientation could be decoded throughout the delay by training and testing within head-upright blocks, and within head-tilted blocks. Importantly, the question of reference frames can be addressed by decoder cross-generalization: If remembered orientations are represented in a retinal reference frame, a decoder trained on head-upright trials would predict a 45º offset in decoded orientation when tested on head-tilted trials (after all, a vertical building becomes diagonal on the retina after head tilt). Conversely, if mnemonic representations are anchored to the real world, no such offset should be observed. Our analysis reveals an early orientation representation in an approximately retinal coordinate frame, while later during the delay the representation approaches a real-world coordinate frame. These results suggest that visual representations can dynamically undergo a reference frame transformation that likely occurs during the transition from perception to working memory, shifting from retinal to real-world anchoring.