Visual Working Memory

Talk Session: Tuesday, May 23, 2023, 5:15 – 7:15 pm, Talk Room 2
Moderator: David Alais, University of Sydney

Schedule of Events | Search Abstracts | Symposia | Talk Sessions | Poster Sessions | Abstract Numbering

Talk 1, 5:15 pm, 55.21

Efficient Measurement of Dynamic Visual Working Memory

Garry Kong¹ (), Isabelle Frisken², Gwenisha J. Liaw², Robert Keys², David Alais²; ¹Waseda University, ²University of Sydney

Here we introduce a novel visual working memory tracking paradigm to measure visual working memory, inspired by continuous psychophysics and motion object tracking. Participants viewed a sequence of stimuli moving along variable paths and were asked to reproduce the path by tracing it on a touchscreen. This reproduction was then compared to the original stimulus to determine error and thus visual working memory performance. Across four experiments, we found that this new method is efficient, reliable and powerful, with only 10 trials per condition required for stable performance estimates, requiring less than 5 minutes of testing. We also demonstrated that the method shows minimal effects from perceptual or attentional confounds. Most importantly, since performance was measured across the trial, this method also allows for the investigation of how visual working memory changes across time. By averaging equivalent time points across trials, we can identify influences from both primacy and recency effects, as well as quantify performance around particularly important points along the motion path. The visual working memory tracking paradigm is therefore especially useful when experimental time is limited, the number of experimental conditions is extensive or when the time-course is the key interest. The method also opens up the study of visual working memory with dynamic stimuli.

Acknowledgements: GK is supported by the Japanese Society for Promotion of Science Kakenhi Grant Number JP22K13874 and Waseda University Grant for Special Research Projects (2022E-040 and 2022C-308). DA is supported by the Australian Research Council (DP210101691).

Talk 2, 5:30 pm, 55.22

No Evidence for a Visual Testing Effect for Novel, Meaningless Objects

Anna C. McCarter¹ (), David E. Huber¹, Rosemary A. Cowell¹; ¹University of Massachusetts at Amherst

The testing effect is a well-established phenomenon in which memory is better for learned information that has been retrieved in a practice test than for learned information that is restudied. Interestingly, the testing effect has almost exclusively been studied using verbal content. Is the testing effect a general property of learning, or does it occur only for verbal information? To investigate this question, we created novel, abstract visual objects composed of a shape and fill pattern. Following initial study of all objects, half of the objects were restudied while the other half underwent recall practice. Recall practice involved presenting one feature (just shape or fill pattern) and having participants visualize the corresponding feature. Then participants either selected the feature they were imagining out of two options (Exp. 1-9) or they drew the feature they were imagining (Exp. 10) followed by feedback. After the re-study/recall practice phase, all objects were tested with final recall and recognition tests. Different experiments manipulated the retention interval, type of feedback, form of restudy, and amount of recall practice, and yet in no case was performance significantly better following recall practice than following restudy. This unexpected result suggests that the testing effect is not a universal process. Two possibilities could explain this finding – either the participants were unable to precisely and accurately recall the visual information, or visual recall was accurate but failed to elicit the expected testing effect. If participants are failing to accurately recall the information, this could lead to either strengthening of incorrect memories or failure to strengthen the correct information. Alternatively, perhaps recall is accurate, but the lack of semantic content precludes any test benefit. Overall, our results suggest that there is no testing effect for purely visual material and that semantic content may be essential to producing a testing effect.

Acknowledgements: This work was supported by NIH grant 1RF1MH114277-01 and University of Massachusetts at Amherst Graduate School Predissertation Grant.

Talk 3, 5:45 pm, 55.23

Nature of the memory trace left by the previous trial in an interceptive task

Esaú Sirius Ventura Pupo¹ (), Raymundo Machado de Azevedo Neto², André Mascioli Cravo¹; ¹Federal University of ABC (UFABC), São Paulo, Brazil, ²Brain Institute, Hospital Israelita Albert Einstein, São Paulo, Brazil

Visual stimuli in the current trial tend to be reproduced more closely to the previous trial, an effect termed serial dependence (SD). Most studies show SD for stimuli when only one of its features is task-relevant. However, in interceptive actions, speed, time to contact, and path length are inherently correlated and could each influence performance in the subsequent trial. We investigated which of these features caused SD using three conditions. Participants (n = 24) performed an online task where they had to press a button while a red square target moving at a constant speed hit a vertical barrier. Blocks could have a single constant distance, speed, or time, with the other two features varying in five levels. We measured the temporal error as the difference between times of interception and response. Each participant's temporal errors were fed into multiple linear regressions with current and previous trial distance, speed, or time as regressors, depending on the condition. The previous trial coefficient was significantly different from zero in a one-sample two-tailed t-test in all conditions, suggesting that none of these features are solely responsible for the bias. Then, we fitted a second model to the previous trial coefficients, using the variables that varied or remained constant as regressors in each block. We found that only the speed and time terms were significantly different from zero in a one-sample two-tailed t-test, indicating that speed and time variations contribute towards SD, whereas the distance does not, even when it is informative. Further experiments are needed to determine whether speed and time are directly stored between trials or both influence a third unstudied variable that could cause SD.

Acknowledgements: This project received support from the São Paulo Research Foundation (FAPESP) through a doctorate's scholarship (2019/06423-7).

Talk 4, 6:00 pm, 55.24

Sequential neuronal spikes as units of cortical coding across visual perception and working memory

Weizhen Xie¹ (), Kareem Zaghloul; ¹National Institute of Neurological Disorders and Stroke, National Institutes of Health

The formation of a visual percept and its retention over a short delay entail coordinated neuronal activity generated from cortical circuits. For instance, past research suggests that the number of spikes within a population of neurons can convey information about a visual item, while the variability in firing rate across neurons constrains the perceptual and mnemonic precision of this item. Conventionally, because this rate-coding scheme is often assumed to follow a Poisson process, the exact timing of spikes is considered inconsequential. Here, we test against this widely-accepted assumption by directly recording from the human temporal lobe using microelectrode arrays as participants performed a retro-cue working memory task. On each trial, participants tried to remember and then later recall one out of two previously presented facial stimuli indicated by a retrospective cue. This design allows us to better isolate and then examine the relationship between perceptually and mnemonically evoked neuronal activity. As the stimuli used in the current study were randomly sampled from a computer-generated circular face spectrum, we also modeled participants’ mnemonic precision based on their recall errors across trials. We find that the onset of facial stimuli evokes temporally organized packets of population activity lasting ~150 ms. The structure of these packets is partially stereotypical, with the variation in spike timing across neurons (hence sequence) conveying information about the identity of a face. Similar packets also occur during the retention interval following the retrospective cue. Furthermore, the consistency in spiking sequences between perceptually and mnemonically evoked neuronal activity is predictive of participants’ subsequent recall precision. Together, these results suggest that temporally organized packets of neuronal activity may constitute basic building blocks of cortical coding across visual perception and memory. As these findings cast some doubts on pure rate-based models, our data highlight a novel neuronal mechanism for visual cognition.

Talk 5, 6:15 pm, 55.25

Parietal impact on visual working memory representation in occipito-temporal cortex

Yaoda Xu¹ (); ¹Yale University

The content of visual working memory (VWM) can be decoded robustly in both human occipito-temporal cortex (OTC) and posterior parietal cortex (PPC). Further work revealed that VWM signal in OTC is largely sustained by feedback from associative areas such as PPC. This raises an intriguing question: What information is retained in OTC during VWM delay? Does it reflect the sustained sensory information initially encoded in OTC or feedback information received from regions like PPC? To examine this, this study took advantage of a previous finding showing that object representational structure (measured from fMRI response patterns) differs between the OTC and PPC during perception. If VWM representation reflects the sensory information encoded during perception, then object representational structure should remain distinct between OTC and PPC during VWM delay. To investigate this, in this fMRI study, twelve human participants retained in VWM objects from eight categories, including bodies, cars, cats, chairs, elephants, faces, houses and scissors. The object representational structure correlation between OTC and PPC was found to be greater during VWM delay than during perception, suggesting that object representations become more similar between these two brain regions during VWM delay. Furthermore, the object representational structure correlation between perception and VWM delay within a region was greater in PPC than OTC, showing that object representation undergoes a greater amount of change in OTC than PPC between perception and VWM delay. Lastly, the object representational structure from OTC during VWM delay was more correlated with that of PPC during perception than with itself during perception, further indicating that OTC object representation during VWM delay becomes more aligned with those of PPC during perception. Together, these results show that the content of VWM representation in OTC is determined more by the feedback signal from PPC than by the sensory information initially encoded in OTC.

Acknowledgements: This work is supported by NIH grant R01EY030854.

Talk 6, 6:30 pm, 55.26

Action consequences guide visual working memory use

Andre Sahakian¹ (), Surya Gayet¹, Chris Paffen¹, Stefan Van der Stigchel¹; ¹Utrecht University, the Netherlands

Visual working memory (VWM) is a store for temporary maintenance of visual information. It is often disregarded, though, that this information is typically stored to enable actions, and therefore, the context of such actions is of great importance for how VWM is used. The severity of the consequence of an action might, for example, affect the precision with which action-relevant information is stored. Here we set out to examine whether strategy changes in VWM-use occur when incorrect actions are penalized. We employed an (online) copying task, where participants recreated a model comprised of several items in a grid, using a (larger) pool of items. Crucially, we manipulated the severity of the penalty: in the high error cost condition participants had to wait 5 seconds after an erroneous item-placement (versus 0.5 seconds in the low error cost condition). Additionally, we manipulated the accessibility of task-relevant information, a well-studied manipulation in this paradigm (implemented here as a 0.5 versus 5 second wait to inspect the model). Manipulating the cost of sampling information provided a direct comparison for the effects of error cost. Our results showed that (1) the number of model inspections halved with higher sampling cost, but were unaffected by error cost; (2) inspection durations increased with higher sampling cost, but were again unaffected by error cost; and (3) the number of errors increased with higher sampling cost, but decreased with higher error costs. Thus, more severe action consequences (error costs) increase the reluctance to act on uncertain information in VWM; but (against our expectations) do not lead to longer, nor to more frequent attempts to store information in VWM. We conclude that, in contrast to the accessibility of information, action consequences do not affect how information is stored, but do affect the willingness to act based on the available information.

Acknowledgements: This project was supported by an ERC Consolidator Grant [grant number ERC-CoG-863732] to Stefan Van der Stigchel, and a Veni grant from Netherlands Organisation for Scientific Research (NWO) [grant number: Vl.Veni.191G.085] to Surya Gayet.

Talk 7, 6:45 pm, 55.27

A model of composing working memories from hierarchical representations acquired through visual learning

Brad Wyble¹ (), Ryan O'Donnell¹, Shekoo Hedayati¹; ¹Penn State University

A hallmark capability of working memory (WM) is the ability to rapidly encode approximations of sensory input with a single brief exposure and then to make that stored information available to other cognitive systems. While WM is often studied with simple features that are bound into objects, a different view of this capacity is that memories are constructed from different levels of a representational hierarchy to most efficiently capture information about a stimulus. We apply this idea using a generative, neurocomputational model, Memory for Latent Representations, that describes how visual learning creates a hierarchy of latent spaces from which memories can be constructed. This model allows for rapid tuning of representations according to task demands and also illustrates how memories of novel stimuli can be created without even forewarning participants of the kind of information, or the format of the response. We demonstrate the flexibility of such a model in storing information of various types, how memory for one object can consist of several traces from different levels of abstraction, and how the format of memory can be rapidly adjusted in the face of new task demands. Empirical demonstrations in human observers support the model by showing that memory traces can be calibrated against task demands by adding or removing detail information for both shape and color attributes. Perhaps most significantly, this perspective challenges current interpretations of working memory tasks by showing that different levels of the hierarchy could be used to store memories in different trials depending on factors such as set size, or stimulus duration. For example if duration is too short to allow binding of features for each of the objects, alternative encoding strategies might encode the whole display as one object and allow for binding after stimulus offset, though at a cost of more rapid forgetting.

Acknowledgements: This work was supported by NSF Grant #1734220.

Talk 8, 7:00 pm, 55.28

Individual variation in optimal encoding strategy in visual working memory

Yin-ting Lin¹ (), Andrew B. Leber¹; ¹The Ohio State University

What determines individual differences in working memory (WM)? Previous research has suggested that WM performance is largely explained by cognitive abilities such as attentional control. However, it is less clear how strategy use accounts for variation in WM performance. Here we explored factors that underlie individual differences in the optimal encoding strategy. In Experiment 1, two displays alternated repeatedly in a flicker paradigm until participants localized a changing item (target). Importantly, there were two targets (one red and one blue), and participants only had to report one target. The ratio of red to blue items varied on each trial. Therefore, to maximize speed, the optimal encoding strategy is to select/encode items in the smaller colour subset. Although choosing the optimal (small subset) target led to faster performance, there were large individual differences in strategy, with some participants frequently reporting the optimal target, and some reporting this target half of the time. Results further showed no relationship between optimal strategy use and visual WM capacity. Experiment 2 tested whether explicit awareness explains strategy use. We found that providing explicit information on the optimal strategy led to a sudden, large increase in optimal target choice. Open-ended reports also showed that participants who were aware of the optimal strategy were more optimal. Experiment 3 further examined whether increasing task difficulty is sufficient to motivate the optimal strategy. In the “one-shot” variant of our paradigm, the displays were presented only once, thus placing a premium on using the optimal strategy to achieve high accuracy. However, strategy use was still sub-optimal; participants did not always selectively encode the small subset. Taken together, results suggest that strategy and ability produce distinct contributions to task performance. Moreover, explicit knowledge, but not greater task demands, leads to a shift to the optimal encoding strategy.

Acknowledgements: This work was supported by NSF BCS-2021038 to ABL