Visual Memory: Working and neural mechanisms

Talk Session: Tuesday, May 21, 2024, 2:30 – 4:15 pm, Talk Room 1
Moderator: Thomas Sprague, UC Santa Barbara

Schedule of Events | Search Abstracts | Symposia | Talk Sessions | Poster Sessions

Talk 1, 2:30 pm, 54.11

The Neural Basis of Visual Working Memory of Real-World Object

Wanru Li¹ (), Jia Yang¹, PingLei Bao¹; ¹Peking University

Sustaining visual information in mind temporally, denoted as visual working memory (VWM), is a core ability of many cognitive functions, such as decision making and problem solving. While existing evidence focusing on low-level features suggests that VWM recruits analogous regions utilized in perceptual tasks, the neural basis of real-world objects within VWM remains elusive. To address this, we investigated the representation of twenty real-world objects across three VWM and three perception tasks using fMRI. In experiment 1, twelve participants performed a retro-cue sequential VWM task, viewing two objects and recalling the cued object after a ten-second delay. We found the identity of the cued objects could be decoded from the lateral occipital complex (LOC) and intraparietal sulcus (IPS), but not in early visual areas. Moreover, representational similarity analysis (RSA) revealed a common representational pattern between VWM and perceptual tasks exclusively in the LOC, indicating that task-relevant high-level visual areas are specifically recruited for VWM maintenance. Experiment 2 leveraged the contralateral bias effect to assess the extent of shared representational properties between VWM and perception. Six participants performed a retro-cue spatial VWM task where they memorized a cued object from two objects presented in separate visual fields. Interestingly, while a strong contralateral bias was confirmed in the perception task, this bias was significantly reduced in the VWM task, evident through an enhanced ipsilateral representation of the cued object. Experiment 3 delayed the retro-cue to the end of the delay, resulting in persistent but significantly reduced involvement of the ipsilateral LOC. Additionally, searchlight RSA revealed larger cortical representational areas in VWM than perception tasks. These results underscore the essential role of both the LOC and IPS in maintaining real-world object representations in VWM, while the sensory-based object representations in high-level visual areas may go beyond the feedforward visual information flow during VWM.

Acknowledgements: National Science and Technology Innovation 2030-Major Project 2022ZD0204803, Natural Science Foundation of China Grant NSFC 32271081,32230043 to P. B. Natural Science Foundation of China Grant NSFC32200857, China Postdoctoral Science Foundation Grant 2023M740125, 2022T150021, 2021M700004 to J.Y.

Talk 2, 2:45 pm, 54.12

fMRI decoding reveals independence in object representations in visual working memory

Yaoda Xu¹ (); ¹Yale University

Often time in everyday visual perception, we need to retain multiple visual objects together in visual working memory (VWM). Yet recent fMRI decoding studies of VWM have predominately focused on the retention of a single object in human occipitotemporal cortex (OTC) and posterior parietal cortex (PPC). How are multiple objects represented together in VWM in these brain regions? Are they represented in an orthogonal and thus independent manner? Or are they coded interactively? To address this, we asked 12 human participants to retain two target objects in VWM. We trained a linear classifier to decode the fMRI response patterns of a pair of target objects A and B when each was retained with object C (i.e., decoding AC vs BC) and tested the classifier’s decoding performance for the same object pair either in the same condition (within-decoding, decoding AC vs BC) or when each was retained with object D (cross-decoding, decoding AD vs BD). Across OTC and PPC, we found no drop in cross-decoding compared to within-decoding during VWM delay, indicating that the two objects in VWM are represented in an orthogonal manner. Such a representational scheme enables independence in VWM representation, effectively preventing interference between the different target objects during VWM retention. Interestingly, during VWM encoding, a cross-decoding drop was observed in OTC (but not in PPC), indicating that an object’s representation is modulated by the identity of another object during encoding in this brain region. However, such a modulation appears to dissipate over the course of VWM retention, likely through feedback mechanims from brain regions such as PPC. Together, these results show independence in target object representations in VWM in the human OTC and PPC, and the emergence of such representations from perception to VWM.

Acknowledgements: This research is supported by NIH grant 1R01EY030854.

Talk 3, 3:00 pm, 54.13

Introspection of relative uncertainty of neural working memory representations in human cortex

Yanming (Alison) li¹ (), Daniel Thayer, Thomas Sprague; ¹University of California, Santa Barbara

Working memory (WM) is the ability to maintain and manipulate a limited amount of information over a short time. Previous research has shown that noisy activation patterns in retinotopic cortex during a WM delay period encode both the memorized feature and the uncertainty regarding how accurately it is represented (Li et al., 2021; Geurts et al, 2022), and that participants can accurately introspect which of several representations they can report most precisely (Fougnie et al, 2012; Suchow et al, 2017; Li & Sprague, 2023). However, how participants read out the relative uncertainty for multiple WM representations from neural activity patterns remains unclear. Here, we acquired fMRI data during a memory guided saccade task in which participants remembered 1 or 2 locations over a 12 s delay period and reported the location of one item with a saccade. Extending our previous study (Li & Sprague, 2023), on each trial, at the end of the delay period participants were either instructed to report a cued item or choose the item they believed they remembered best with a saccade. After the memory report, participants reported their uncertainty about the reported location by adjusting the extent of an arc (as in Li et al, 2021). Results showed that when participants were asked to report their best-remembered item, recall error and uncertainty were both lower compared to randomly-cued trials, consistent with accurate introspection of the relative quality of multiple WM representations. Moreover, delay-period WM representations reconstructed from activation patterns in extrastriate cortex were stronger for memory items reported on “report best” trials as compared to the non-reported item. These findings suggest that participants can simultaneously compare and report the quality of multiple remembered locations, and demonstrate that these reports are based on the quality of neural WM representations in retinotopic cortex.

Acknowledgements: Research was sponsored in part by the U.S. Army Research Office and accomplished under contract W911NF-19-D-0001 for the Institute for Collaborative Biotechnologies, an Alfred P Sloan Foundation Research Fellowship, and a University of California, Santa Barbara Academic Senate Research Grant

Talk 4, 3:15 pm, 54.14

Flexible memory interplays: selective reactivation of long-term memories in working memory

Eren Günseli¹ (), Duygu Yücel¹, Nursena Ataseven¹, Yağmur Damla Şentürk¹, Nursima Ünver¹, Şahcan Özdemir¹, Lara Todorova¹, Berna Güler¹, Betül Türk¹, Can Demircan¹, Keisuke Fukuda², Christian N L Olivers³, Tobias Egner⁴; ¹Sabanci University, ²University of Toronto, ³Vrije Universiteit Amsterdam, ⁴Duke University

Working memory is defined as the online storage space for ongoing tasks. It stores both newly encoded information and retrieved long-term memories. However, there is a growing amount of work to suggest that long-term memories can also guide behavior. This raises the question: Why do humans invest metabolic resources in reactivating long-term memories in working memory instead of guiding behavior directly via long-term memory? We conducted six experiments examining working memory reactivation of long-term memories in anticipation of task demands encompassing protection against interference, behavioral guidance, and adaptation to novel settings. Using behavioral and electrophysiological indices, we measured the extent to which long-term memories are reactivated in working memory in anticipation of these task demands relative to the anticipation of a recognition task, which constituted a baseline. Compared to this baseline, we found equal memory reactivation when anticipating perceptual interference and dual-task interference, and less memory reactivation when anticipating attentional guidance. On the other hand, reactivation was stronger for task switching, contextual changes, and performing mental operations. These results suggest that the reactivation of long-term memories in working memory is not primarily for protection against interference or behavioral guidance. Instead, stronger reactivation occurs when there is a need to update the memories themselves (i.e., perform a mental operation) or the settings in which they are used (i.e., the task rules and the context). This insight implies that the goal of reactivating long-term memories in working memory may be to facilitate adaptation to novel situations. Our research challenges influential memory models and recent empirical work that consider working memory as the default buffer for retrieved long-term memories and instead highlights a flexible and dynamic interplay between long-term memories and working memory.

Talk 5, 3:30 pm, 54.15

Associative learning changes multivariate neural signatures of visual working memory

William Ngiam^1,2 (), William Thyer^1,2, Henry Jones^1,2, Darius Suplica^1,2, Will Epstein^1,2, Edward Awh^1,2; ¹Department of Psychology, University of Chicago, ²Institute of Mind and Biology, University of Chicago

A hallmark of visual working memory is its sharp capacity limit, though this limit can be circumvented using learned knowledge. For example, when arrays of to-be-remembered items contain statistical regularities, people can learn the associations between items and recall more information overall (Brady et al., 2009; Ngiam et al., 2019). One proposed mechanism for how this recall benefit is achieved is through ‘memory compression’ – redundancies introduce a reduction of information per item, enabling more items to be stored online. Another proposed mechanism is that pointers are efficiently allocated to each ‘chunk’ with the benefit coming from long-term memory retrieval rather than changes to working memory itself. In an attempt to distinguish between these possibilities, we turned to an EEG measure that tracks the number of individuated items stored in working memory (mvLoad; Thyer et al., 2022). The memory compression account predicts an overall increase in the number of items stored online, whereas the long-term memory retrieval account predicts a reduction in working memory load. Subjects completed a training session where they learned specific color-color pairs. In a subsequent EEG session, subjects completed a recall task with 2 random colors, 4 random colors, or 2 learned color pairs. mvLoad analysis showed a reduction in working memory load for the 2 learned pairs condition (from 4 towards 2), consistent with the notion that an item-based pointer is assigned to each chunk. Moreover, multidimensional scaling shows an additional independent signal that distinguishes the 2 learned pairs condition from the other conditions. We propose that this additional signal reflects the involvement of long-term memory, consistent with the notion that the learned association is being relied upon to maintain the information.

Acknowledgements: This research was supported by the National Institute of Health R01-MH087214 grant awarded to Edward Awh and Edward Vogel.

Talk 6, 3:45 pm, 54.16

Sustained stimulus-selective multi-unit activity in human primary visual cortex

Lucas Nadolskis¹ (), Galen Pogoncheff¹, Jacob Granley¹, Alfonso Rodil², Leili Soo², Lily Turkstra¹, Thomas Sprague¹, Arantxa Alfaro Saez^2,3, Cristina Soto Sanchez², Eduardo Fernandez Jover², Michael Beyeler¹; ¹University of California - Santa Barbara, ²Universidad Miguel Hernandez de Elche, Spain, ³Section of Neurology, Hospital Vega Baja, Orihuela, Spain

Introduction: Neural activity related to visual working memory (WM) has been found in various cortical areas of primates and humans. However, the role of multi-unit activity in human V1 during WM tasks is not fully understood. We explored this by examining intracortical recordings from an awake blind human with a visual prosthesis (Utah array in parafoveal V1) during a delayed-response WM task. Methods: In 90 trials, one of three chosen electrodes stimulated a visual percept (phosphene) in the participant (stimulation period), who then had to remember its shape and location for 3 or 5 seconds (delay period), followed by an auditory cue and a recall period, during which the participant was asked to intently visualize the remembered phosphene. Neural activity was recorded and analyzed for multi-unit activity (MUA), entire spiking activity (ESA), and local field potential (LFP). Results: Significant differences in MUA, ESA, and LFP (theta, alpha, and beta bands) were observed across different trial periods (stimulation, delay, recall, and spontaneous; t-tests, p<.05). Each electrode's neural signature was distinct during delay and recall (over 90% accuracy in leave-one-trial-out cross-validation), with day-to-day drifts. The directions of maximum variability in the recall period neural activity formed a representative neural basis for each electrode, and enabled classification with a random forest classifier for both delay (97% accuracy) and stimulation period activities (88% accuracy). These shared signatures could be learned from the delay or recall period activity, but not from the electrically evoked activity, suggesting that WM elicits a subset of the full activity present during electrical stimulation. Conclusion: Our findings underscore V1's crucial role in retaining information at the neuronal level over delay periods. The transformation of representations during the recall period suggests that the encoded information is more abstract than the sensory activity evoked during stimulation.

Acknowledgements: NIH DP2-LM014268, Alfred P. Sloan Foundation Research Fellowship, PDC2022-133952-100 and PID2022-141606OB-I00 from the Spanish “Ministerio de Ciencia, Innovación y Universidades”, European Union’s Horizon 2020 Research and Innovation Programme under Grant Agreement No. 899287 (NeuraViPeR).

Talk 7, 4:00 pm, 54.17

Linking behavioral and neural estimates of trial-by-trial working memory information content

Ying Zhou^1,2 (), Clayton Curtis², Daryl Fougnie¹, Kartik Sreenivasan^1,3; ¹Division of Science and Mathematics, New York University Abu Dhabi, ²Department of Psychology, New York University, ³Center for Brain and Health, New York University Abu Dhabi

How is working memory (WM) information represented in the brain? Neural and computational models have used data aggregated over hundreds of trials to argue for different perspectives on how neural activity encodes individual memories. The two main perspectives are information rich representations such as in probabilistic coding models (a probability distribution over the whole feature space), and information sparse representations, such as in high-threshold ( a precise feature value) or drift models (a value with a confidence interval unrelated to the direction of drift). The use of aggregate data represents a key inferential bottleneck that critically limits the ability to adjudicate between different formats of individual memory coding in WM. This study used a powerful method to link behavioral and neural estimates of WM representation on individual trials. We asked participants (n = 12) to memorize a motion direction over a brief delay. After the delay, instead of making a single report about the memorized direction, they indicated their memory by placing 6 “bets”, resulting in a distribution over the 360° direction space that reflected their probabilistic memory representation on individual trials. Additionally, we used a Bayesian decoder to estimate the posterior of the memorized direction given the fMRI signal during memory maintenance on individual trials. Comparing the shape of the behavioral and neural estimates on individual trials, we found a significant correspondence in their width in occipital, parietal and frontal regions (ps < .007; Cohen’s ds > .767), and critically, a significant correspondence in their asymmetry in early visual cortex (p < .001; Cohen’s d = .779). These results indicate (1) individual WM representations are complex probability distributions that contain more information than that can be deduced from aggregate data; (2) early visual cortex contains richer information about WM than other brain regions, with meaningful asymmetry information influencing behavior.

Acknowledgements: NYUAD Research Institute grant CG012