VSS, May 13-18

Visual Memory: Working, objects, features

Talk Session: Sunday, May 15, 2022, 5:15 – 7:15 pm EDT, Talk Room 1
Moderator: Timothy Brady, UCSD

Times are being displayed in EDT timezone (Florida time): Wednesday, July 6, 3:33 am EDT America/New_York.
To see the V-VSS schedule in your timezone, Log In and set your timezone.

Search Abstracts | VSS Talk Sessions | VSS Poster Sessions | V-VSS Talk Sessions | V-VSS Poster Sessions

Talk 1, 5:15 pm, 35.11

Long-term memory enhances object retention in visual working memory independently from perceptual complexity

Markus Conci1 (), Hermann J. Müller1; 1Ludwig-Maximilians-Universität München

Visual working memory (VWM) is typically found to be severely limited in capacity, but this limitation may be ameliorated by providing familiar objects that are associated with knowledge stored in long-term memory. However, comparing meaningful and meaningless stimuli usually entails a confound, because different types of objects also tend to vary in terms of their inherent perceptual complexity. Here, we present two sets of findings that aim to demonstrate that object meaning can benefit VWM independently from variations in perceptual complexity. For instance, in study 1, we demonstrate that simple (unicolored) objects were remembered better than more complex, yet meaningless color-shape configurations. However, perceptually identical, complex but meaningful items revealed a reliable VWM performance benefit relative to the meaningless configurations (with the capacity estimates for meaningful objects in fact being comparable to the simple, unicolored objects). This demonstrates that the effects of meaning upon object memory may be dissociated from concurrent variations of perceptual complexity. In addition, study 2 demonstrates that actively learning the meaning of previously meaningless objects can improve VWM capacity beyond basic stimulus repetition effects. For instance, learning the meaning of Chinese characters enhanced VWM performance, relative to a second set of characters that were just viewed passively (without providing their meaning). This shows that learning generates meaning, which in turn enhances VWM performance. Together, these findings show that the short-term retention of an object depends on object knowledge, where knowledge may be derived from associations about an object’s meaning in long-term memory thereby enhancing the representation of an object that is currently maintained.

Talk 2, 5:30 pm, 35.12

Dissociating the Impact of Object-Color Expectations and Object-Color Violations on Visual Feature Memory

Kimele Persaud1 (), Elizabeth Bonawitz2; 1Rutgers University - Newark, 2Harvard University

Study events that are consistent with our prior expectations are often better remembered than expectation-unrelated events. Paradoxically, events that violate our expectations are also better remembered. What remains unclear is whether visual memory for expectation-consistent and expectation-violating events are supported by qualitatively different processes. Here we explore whether visual memory for expectation-related events are differentially impacted by mechanisms that underlie recognition and recall processes. We further assessed how the degree of expectation-consistency impacts memory for other features of study events. Across four experiments, we manipulated the degree to which study events adhered to people’s prior expectations (i.e., the color of objects) and then assessed memory (recall and recognition) for expectation-relevant features (i.e., object-color) and expectation-irrelevant features (object-shape). We propose an account that allows for competing mechanisms in memory encoding, storage, and retrieval, helping to explain the paradox in prior studies. Since recognition memory is backed by efficient encoding and storage processes, expectation-consistent events are better recognized because noise in the system is biased towards category expectations in storage. Expectation-violating events are better recognized because expectation-violating events lead to more resources spent on encoding all features of the study event. In contrast, recall is backed by boosted encoding and memory search processes for retrieving stored events, which are also biased toward expectations. Thus, expectation-consistent events are boosted in recall, but not expectation-violating events. Across experiments, we find evidence supporting both the “boosted encoding/storage” and “memory search” mechanisms. Expectation-consistent events were better recognized and recalled, while expectation-violating events were better recognized. Also, the advantage of expectation-violation, but not expectation-consistency, extended to memory for expectation-irrelevant features of the study event. These findings suggest that expectation-consistent and expectation-violating information are qualitatively dissociable in their impact on recognition and recall processes as well as their influence on memory for expectation-irrelevant features of study events.

Acknowledgements: This work was supported by NSF-SPRF 1911656 (KP), NSF SES-1627971 (EB) and the Jacobs Foundation (EB).

Talk 3, 5:45 pm, 35.13

Event segments and sensory memory storage

Shaoying Wang1, Srimant Tripathy2, Haluk Öǧmen1; 1University of Denver, 2University of Bradford

The temporal structure of ecological stimuli contains events delineated by event boundaries. Previously, it was shown that Sensory Memory (SM) for motion is allocated exclusively to the current event-segment (Tripathy & Öǧmen, 2018, Frontiers in Psychology). It is not clear however how SM is reset and which items in the current event are stored in SM. We considered two models: In the “complete-reset model”, an event boundary purges completely the prior contents of SM and stores exclusively only those items that underwent change at the event boundary. In the “partial-reset model”, all items that are present post-boundary are stored in SM and those that underwent change are updated. Our stimuli contained four disks moving along linear trajectories with different directions. Two of the disks changed their directions at 400ms and the remaining two at 800 ms. In Experiment 1, the disks that changed direction at 400ms were removed from the display at the time the other two disks changed their direction (800 ms), whereas in Experiment 2, they remained on the display until the end of the presentation (1200ms). Subjects were instructed to report the directions of cued disk(s) from different event segments with varying cue delays. We used two signatures of SM to assess its contents: Rapid decay with cue-delay and partial-report superiority. The results of Experiment 1 showed that the items that changed at the most recent event boundary were stored in SM. Items from previous event segments were not. In Experiment 2, the results indicated that only items that were present after the event boundary were stored in SM. Taken together, the results support the partial-reset model.

Talk 4, 6:00 pm, 35.14

A Beta-Variational Auto-Encoder Model of Human Visual Representation Formation in Utility-Based Learning

Tyler Malloy1 (), Chris R. Sims1; 1Rensselaer Polytechnic Institute

The human brain is capable of forming informationally constrained representations of complex visual stimuli in order to achieve its behavioural goals, such as utility-based learning. Recently, methods borrowed from machine learning have demonstrated a close connection between the mechanisms of visual representation formation in primate brains with the latent representations formed by Beta-Variational Auto-Encoders (Beta-VAEs). While auto-encoder models capture some aspects of visual representations, they fail to explain how visual representations are adapted in a task-directed manner. We developed a model of visual representation formation in learning environments based on a modified Beta-VAE model that simultaneously learns the task-specific utility of visual information. We hypothesized that humans update their visual representations as they learn which visual features are associated with utility in learning tasks. To test this hypothesis, we applied the proposed model onto the data from a visual contextual bandit learning task [Niv et al. 2015; J. Neuroscience]. The experiment involved humans (N=22) learning the utility associated with 9 possible visual features (3 colors, shapes or textures). Critically, our model takes in as input the same visual information that is presented to participants, instead of the hand-crafted features typically used to model human learning. A comparison of predictive accuracy between our proposed model and models using hand-crafted features demonstrated a similar correlation to human learning. These results show that representations formed by our Beta-VAE based model can predict human learning from complex visual information. Additionally, our proposed model makes predictions of how visual representations adapt during human learning in a utility-based task. Further, we performed a comparison of our proposed model across a range of parameters such as information-constraint, utility-weight, and number of training steps between predictions. Results from this comparison give insight into how the human brain adjusts its visual representation formation during learning.

Acknowledgements: This work was supported by NSF research grant DRL-1915874 to CRS and an IBM AIRC scholarship to TJM.

Talk 5, 6:15 pm, 35.15

Visual working memory in action: Planning for multiple potential actions alongside multi-item visual encoding and retention

Rose Nasrawi1 (), Freek van Ede1,2; 1Institute for Brain and Behavior Amsterdam, Vrije Universiteit Amsterdam, 2Oxford Centre for Human Brain Activity, University of Oxford

Visual working memory allows us to use past visual information to guide upcoming future behavior, serving as a bridge between perception and action. Previous research into visual working memory has classically concentrated on the (neural) mechanisms of encoding and retention of visual information. Yet, in our daily lives, we often rely on visual representations in memory to prepare for and guide potential future actions. Recent studies have demonstrated that, when retaining a single visual item in working memory, planning for the required manual action starts during the memory delay. We asked whether and how multiple potential actions are planned alongside the encoding and retention of multiple visual items during working memory. Human participants performed a visual working memory task with a delayed orientation-reproduction report (of one item), and a memory load manipulation (one/two/four items). Using scalp EEG, we measured 15-25Hz beta activity in central electrodes contralateral to the required response hand – a canonical neural marker of action planning. We show an attenuation of beta activity not only in load one (with high certainty about the prospective action), but also in load two (with two potential prospective actions), compared to load four (with low certainty about the prospective action). Action planning in load two occurs regardless of whether the two potential actions require a similar and dissimilar manual response. Moreover, the degree of beta attenuation during the memory delay in both load one and load two is predictive of the speed of ensuing memory-guided action. These results indicate that multiple potential actions are planned alongside the encoding and retention of multiple visual items in working memory. They bring the concept of parallel action planning to the domain of multi-item visual working memory, and highlight how visual working memory and action planning jointly help us prepare for the potential future.

Acknowledgements: This research was supported by an ERC Starting Grant from the European Research Council (MEMTICIPATION, 850636), a Newton International Fellowship from the Royal Society and the British Academy (NF140330), and a Marie Skłodowska-Curie Fellowship from the European Commission (ACCESS2WM).

Talk 6, 6:30 pm, 35.16

Visual Working Memory Performance With Just 1 Item Predicts Nearly All of the Variance in Performance with 5 Items

Timothy Brady1 (); 1University of California, San Diego

What limits working memory performance when there are many items to be held in mind? Many models of working memory capacity focus on factors that are present primarily at high set sizes (e.g., interference between items; upper bounds on number of items that can be held in mind; etc). These models assume that performance is effectively ‘at ceiling’ when remembering just 1 item, and so little can be learned about working memory from such "sub-capacity" trials. Here we test this by taking an individual differences approach. We use the TCC model (Schurgin et al. 2020) and both continuous report and a specially designed 4-AFC task to measure visual working memory performance at set size 1 and set size 5, and ask how they are related. In 3 studies (all N>100) we find that performance at set size 1 explains 80-95% of the explainable variance in set size 5 performance. By contrast, a challenging mental rotation task explains <40% of the variance at set size 5, suggesting this is specific to working memory capacity. This raises important challenges for models who focus primarily on explaining high set sizes when considering the source of working memory limits (slots, interference, etc). It is most consistent with resource models, particularly those where the same fixed set of resources is given all to a single item at set size 1 or split among all 5 items at set size 5. This work also suggests that the focus on studying working memory 'capacity' only at high set sizes is likely counterproductive: the same capacity seems to be measurable at set size 1, and the largest variation between participants occurs at set size 1, and the largest drop in performance occurs in moving from set size 1 to set size 2.

Acknowledgements: NSF BCS-1653457 to TFB

Talk 7, 6:45 pm, 35.17

Perceptual similarity judgments predict the precision but not the distribution of errors in working memory

Paul Bays1, Ivan Tomić1; 1University of Cambridge

Models based on population coding have provided a parsimonious account of empirical distributions of error in visual working memory (VWM) tasks. Inspired by electrophysiological observations of sensory neurons, these models account for VWM errors and the effects of set size based on two key properties of neural response functions, the tuning width and activity level, which can be directly related to variables of signal detection theory, including d-prime. A new perspective on this class of models has recently been presented in the form of the Target Competition Confusability (TCC) model (Schurgin et al., Nat Hum Behav, 2020). The core claim of TCC is that the distribution of recall errors on e.g. a colour wheel, can be accounted for by the perceptual similarity of values in that feature space, i.e. the perceptual similarity function takes on an equivalent role to population tuning, obviating the need to fit a tuning width parameter to the recall data. Here we set out to verify this claim by testing the correspondence between population coding and TCC components that predict the shape of VWM error distributions. Using four different visual feature spaces, we measured psychophysical similarity and working memory errors in the same participants. The results revealed no consistent relationship between perceptual similarity functions and VWM error distributions at individual or group levels. In contrast, we found strong evidence for a correlation between the variability of similarity judgements in the perceptual task and the activity level (signal-to-noise) in the population coding model fit to VWM errors. Our results suggest that perceptual similarity functions are not predictive of VWM errors, but that a common source of variability affects perceptual difference judgements and recall from VWM, perhaps related to broader cognitive ability.

Acknowledgements: This study was supported by the Wellcome Trust.

Talk 8, 7:00 pm, 35.18

Neural signatures of serial dependence emerge during cued selection in working memory

Cora Fischer1 (), Jochen Kaiser1, Christoph Bledowski1; 1Institute of Medical Psychology, Goethe University Frankfurt, Germany

Visual objects separated in time are not processed independently from each other. Instead, a current object is often reported as more similar to a previously encoded, but now irrelevant object than it actually was. This phenomenon is called serial dependence. Until now, it has remained unclear whether serial dependence occurs during an early stage in the object processing hierarchy, i.e., when an object is encoded into the visual system, or during later stages, i.e., when an object is retained in working memory or selected for an action. To determine at which stage an object representation becomes biased, we recorded neuronal activity using MEG while subjects encoded and memorized two sequentially presented motion directions and, after a short delay, selected one direction for a report based on a retro-cue. Using a model-based MEG decoding approach, we found that the neural representation of a current motion direction was shifted toward the previous motion direction only after the retro-cue, when a direction was selected for subsequent report. On a single-trial level, the shift of the decoded motion direction after the retro-cue predicted the magnitude and direction of subjects’ response errors during continuous recall. These results show that an object representation is susceptible to serial dependence especially during a late stage of object processing, when it changes its format from a representation that is memorized for potential use to a representation that is already selected and prepared for upcoming action.

Acknowledgements: This study was supported by the German Academic Scholarship Foundation (PhD Scholarship awarded to C.F.)