Friday, May 13, 2022, 12:00 – 2:00 pm EDT, Talk Room 2

Organizers: Alex White¹, Kendrick Kay²; ¹Barnard College, Columbia University, ²University of Minnesota
Presenters: Alex L. White, Clare Press, Charlie S. Burlingham, Clayton E. Curtis, Jesse Breedlove

< Back to 2022 Symposia

The concept of sensory representation has been immensely productive for studying visual cortex, especially in the context of ‘image-computable models’ of visually evoked responses. At the same time, many experiments have demonstrated that various forms of attention modulate those evoked responses. Several computational models of attention explain how task-relevant stimuli are represented more faithfully than task-irrelevant stimuli. However, these models still paint an incomplete picture of processing in visual cortex. Activity in visual brain regions has been shown to depend on complex interactions between bottom-up sensory input and task demands. In many cases, that activity is affected by cognitive factors that are not clearly related to sensory representation or attention, such as memory, arousal, and expectation. This symposium will bring together complementary perspectives on cognitive effects on activity in visual cortex. Each speaker will present a recently studied interaction of vision and cognition and how it manifests in experimental data. In addition, the speakers will consider the underlying mechanisms for the effects they observe. Key questions include: Are visual representations simply enhanced for any behaviorally relevant stimulus, or do task-specific neural networks modulate visual cortex only in the presence of specific stimuli? How do we interpret activity observed in the absence of retinal stimulation? Are there distinct representational systems for visual working memory, imagery, and expectations? In a final panel discussion, we will broach additional fundamental issues: To what extent is it possible to study representation in the absence of manipulating cognition? How can we build formal models that account for the range of cognitive and sensory effects in visual cortex? Each of the 5 speakers will be allotted 15 minutes for presentation plus 3 minutes for audience questions, for a total of 18 minutes per speaker. The final panel discussion will last 30 minutes. The panel will be moderated by Kendrick Kay, who will deliver an initial brief summary that will attempt to integrate the disparate studies presented by the speakers and weave together a coherent bigger picture regarding overall challenges and goals for studying cognitive effects on activity in visual cortex. A panel discussion will follow with questions posed by the moderator, as well as questions solicited from the audience.

Presentations

High specificity of top-down modulation in word-selective cortex

Alex L. White¹, Kendrick Kay², Jason D. Yeatman³; ¹Barnard College, Columbia University, ²University of Minnesota, ³Stanford University

Visual cortex is capable of processing a wide variety of stimuli for any number of behavioral tasks. So how does the specific information required for a given task get selected and routed to other necessary brain regions? In general, stimuli that are relevant to the current task evoke stronger responses than stimuli that are irrelevant, due to attentional selection on the basis of visual field location or non-spatial features. We will first review evidence that such attentional effects happen in category-selective regions, such as the visual word form area, as well as early retinotopic regions. We will then demonstrate evidence for top-down effects that are not domain-general, but extremely specific to task demands, stimulus features, and brain region. We measured fMRI responses to written words and non-letter shapes in retinotopic areas as well as word- and face-selective regions of ventral occipitotemporal cortex. In word-selective regions, letter strings evoked much larger responses when they were task-relevant (during a lexical decision task) than when they were irrelevant (during a color change task on the fixation mark). However, non-letter shapes evoked smaller responses when they were task-relevant than when irrelevant. This surprising modulation pattern was specific to word-selective regions, where response variability was also highly correlated with a region in the pre-central sulcus that is involved in spoken language. Therefore, we suggest that top-down modulations in visual cortex do not just generally enhance task-relevant stimuli and filter irrelevant stimuli, but can reflect targeted communication with broader networks recruited for specific tasks.

The influence of expectation on visual cortical processing

Clare Press^1,2, Emily Thomas^1,3, Daniel Yon¹; ¹Birkbeck, University of London, ²University College London, ³New York University

It is widely assumed that we must use predictions to determine the nature of our perceptual experiences. Work from the last few years suggests that supporting mechanisms operate via top-down modulations of sensory processing. However, theories within the domain of action concerning the operation of these mechanisms are at odds with those from other perceptual disciplines. Specifically, action theories propose that we cancel predicted events from perceptual processing to render our experiences informative – telling us what we did not already know. In contrast, theories outside of action – typically couched within Bayesian frameworks – demonstrate that we combine our predictions (priors) with the evidence (likelihood) to determine perception (posterior). Such functions are achieved via predictions sharpening processing in early sensory regions. In this talk I will present three fMRI studies from our lab that ask how these predictions really shape early visual processing. They will ask whether action predictions in fact shape visual processing differently from other types of prediction and about differences in representation across different cortical laminae. The studies compare processing of observed avatar movements and simple grating events, and ask about the information content associated with the stimulus types as well as signal level across different types of voxels. We can conclude that action expectations exhibit a similar sharpening effect on visual processing to other expectations, rendering our perception more veridical on average. Future work must now establish how we also use our predictions – across domains – to yield informative experiences.

Task-related activity in human visual cortex

Charlie S. Burlingham¹, Zvi Roth², Saghar Mirbagheri³, David J. Heeger¹, Elisha P. Merriam²; ¹New York University, ²National Institute of Mental Health, National Institutes of Health, ³University of Washington

Early visual cortex exhibits widespread hemodynamic responses during task performance even in the absence of a visual stimulus. Unlike the effects of spatial attention, these “task-related responses” rise and fall around trial onsets, are spatially diffuse, and even occur in complete darkness. In visual cortex, task-related and stimulus-evoked responses are similar in amplitude and sum together. Therefore, to interpret BOLD fMRI signals, it is critical to characterize task-related responses and understand how they change with task parameters. We measured fMRI responses in early visual cortex (V1/2/3) while human observers judged the orientation of a small peripheral grating in the right visual field. We measured task-related responses by only analyzing voxels in the ipsilateral hemisphere, i.e., far from the stimulus representation. Task-related responses were present in all observers. Response amplitude and timing precision were modulated by task difficulty, reward, and behavioral performance, variables that are frequently manipulated in cognitive neuroscience experiments. Surprising events, e.g., responding incorrectly when the task was easy, produced the largest modulations. Response amplitude also covaried with peripheral signatures of arousal, including pupil dilation and changes in heart rate. Our findings demonstrate that activity in early visual cortex reflects internal state — to such a large extent that behavioral performance can have a greater impact on BOLD activity than a potent visual stimulus. We discuss the possible physiological origins of task-related responses, what information about internal state can be gleaned from them, and analytic approaches for modelling them.

Unveiling the abstract format of mnemonic representations

Clayton E. Curtis¹, Yuna Kwak¹; ¹New York University

Working memory (WM) enables information storage for future use, bridging the gap between perception and behavior. We hypothesize that WM representations are abstractions of low-level perceptual features. Yet the neural nature of these putative abstract representations has thus far remained impenetrable. Here, we first demonstrate that distinct visual stimuli (orientated gratings and moving dots) are flexibly re-coded into the same WM format in visual and parietal cortex when that representation is useful for memory-guided behavior. Next, we aimed to reveal the latent nature of the abstract WM representation. We predicted that the spatial distribution of higher response amplitudes across a topographic map forms a line at a given angle, as if the retinal positions constituting a line were actually visually stimulated. To test this, we reconstructed the spatial profile of neural activity during WM by projecting the amplitudes of voxel activity during the delay period for each orientation and direction condition into visual field space using parameters obtained from models of each visual map’s population receptive field. Remarkably, the visualization technique unveiled a stripe encoded in the amplitudes of voxel activity at an angle matching the remembered feature in many of the visual maps. Finally, we used models of V1 that demonstrate the feasibility of such a working memory mechanism and ruled out potential confounds. We conclude that mnemonic representations in visual cortex are abstractions of percepts that are more efficient than and proximal to the behaviors they guide.

With or without the retina: analyses of non-optic visual activity in the brain

Jesse Breedlove¹, Ghislain St-Yves¹, Logan Dowdle¹, Tom Jhou², Cheryl Olman¹, Thomas Naselaris¹; ¹University of Minnesota, ²Medical University of South Carolina

One way to investigate the contribution of cognition on activity in the visual cortex is to fix or remove the retinal input altogether. There are many such non-optic visual experiences to draw from (e.g., mental imagery, synesthesia, hallucinations), all of which produce brain activity patterns consistent with the visual content of the experience. But how does the visual system manage to both accurately represent the external world and synthesize visual experiences? We approach this question by expanding on a theory that the human visual system embodies a probabilistic generative model of the visual world. We propose that retinal vision is just one form of inference that this internal model can support, and that activity in visual cortex observed in the absence of retinal stimulation can be interpreted as the most probable consequence unpacked from imagined, remembered, or otherwise assumed causes. When applied to mental imagery, this theory predicts that the encoding of imagined stimuli in low-level visual areas will resemble the encoding of seen stimuli in higher areas. We confirmed this prediction by estimating imagery encoding models from brain activity measured while subjects imagined complex visual stimuli accompanied by unchanging retinal input. In a different fMRI study, we investigated another far rarer form of non-optic vision: a case subject who, after losing their sight to retinal degeneration, now “sees” objects they touch or hear. The existence of this phenomenon further supports visual perception being a generative process that depends as much on top-down inference as on retinal input.

< Back to 2022 Symposia

Friday, May 13, 2022, 12:00 – 2:00 pm EDT, Talk Room 1

Organizers: Chaz Firestone¹, Alon Hafri¹; ¹Johns Hopkins University
Presenters: Alon Hafri, Melissa Le-Hoa Võ, Liuba Papeo, Daniel Kaiser, Hongjing Lu

< Back to 2022 Symposia

A typical VSS program devotes sections to low-level properties such as motion, orientation, and location; higher-level properties such as faces, scenes, and materials; and core visual processes such as working memory and attention. Yet a notable absence among these are relational representations: properties holding *between* elements, beyond any properties each element has on its own. For example, beyond perceiving red apples and glass bowls, we may also see apples contained inside bowls; beyond perceiving an object and its motion, we may see it collide with another object; and beyond perceiving two agents, we may also see them socially interact. The aim of this symposium is to showcase work that investigates relational representation using the methods and tools of vision science, including classic paradigms from visual cognition, modern neuroimaging techniques, and state-of-the-art computational modeling. A central theme is that fully understanding the nature of visual perception — including core processes such as object and scene representation, visual attention, and working memory — requires a consideration of how visual elements relate to one another. First, Alon Hafri and Chaz Firestone will provide an overview of the “relational landscape”. They will delineate criteria for determining whether a relational property is perceived rather than merely judged or inferred, and they will discuss several case studies exemplifying this framework. Second, Melissa Võ will discuss her work on “scene grammar”, whereby the mind represents natural environments in terms of the typical composition of their objects (e.g., soap generally appears on sinks). Võ suggests that certain clusters of objects (especially “anchor objects”) guide visual search, object perception, and memory. Third, Liuba Papeo will present her work on social relations (e.g., when two agents approach, argue, or fight). Papeo shows that the visual system identifies social relations through a prototypical “social template”, and she explores the ways such representations generalize across visual contexts. Fourth, Daniel Kaiser will extend the discussion from objects to scene structure. Using neuroimaging evidence, he shows that natural scene processing is fundamentally relational: when configural relations between scene parts are disrupted, there are downstream consequences for scene and object processing. Finally, Hongjing Lu and Phil Kellman will discuss the computational machinery necessary to achieve relational representations. Although deep-learning models achieve remarkable success at many vision tasks, Lu and Kellman present modeling evidence arguing that abstract structure is necessary for representing visual relations in ways that go beyond mere pattern classification. Overall, this work explores how relational structure plays a crucial role in how we see the world around us, and raises important questions for future vision science research. David Marr famously defined vision as the capacity to “know what is where by looking” — to represent objects and their features, located somewhere in space. The work showcased here adds an exciting dimension to this capacity: not only what and where, but “how” visual elements are configured in their physical and social environment.

Presentations

Perceiving relational structure

Alon Hafri¹, Chaz Firestone¹; ¹Johns Hopkins University

When we open our eyes, we immediately see the colors, shapes, and sizes of the objects around us — round apples, wooden tables, small kittens, and so on — all without effort or intention. Now consider relations between these objects: An apple supported by a table, or two kittens chasing one another. Are these experiences just as immediate and perceptual, or do they require effort and reflection to arise? Which properties of relations are genuinely perceived, and how can we know? Here, we outline a framework for distinguishing perception of relations from mere judgments about them, centered on behavioral “signatures” that implicate rapid, automatic visual processing as distinct from high-level judgment. We then discuss several case studies demonstrating that visual relations fall within this framework. First, we show that physical relations such as containment and support are extracted in an abstract manner, such that instances of these relations involving very different objects are confused for one another in fast target-identification tasks. Second, we show that the mind “fills in” required elements of a relation that are inferred from physical interaction (e.g., a man running into an invisible “wall”), producing visual priming in object detection tasks. Third, we show that when objects look like they can physically fit together, this impression influences numerosity estimates of those objects. We argue that visual processing itself extracts sophisticated, structured relations, and we reflect on the consequences of this view for theorizing about visual perception more broadly.

Hierarchical relations of objects in real-world scenes

Melissa Le-Hoa Võ¹; ¹Goethe University – Frankfurt

The sources that guide attention in real-world scenes are manifold and interact in complex ways. We have been arguing for a while now that attention during scene viewing is mainly controlled by generic scene knowledge regarding the meaningful composition of objects that make up a scene (a.k.a. scene grammar). Contrary to arbitrary target objects placed in random arrays of distractors, objects in naturalistic scenes are placed in a very rule-governed manner. In this talk, I will highlight some recent studies from my lab in which we have tried to shed more light on the hierarchical nature of scene grammar. In particular, we have found that scenes can be decomposed into smaller, meaningful clusters of objects, which we have started to call “phrases”. At the core of these phrases you will find so-called “anchor objects”, which are often larger, stationary objects that anchor strong relational predictions about where other objects within the phrase are expected to be. Thus, within a “phrase” the spatial relations of objects are strongly defined. Manipulating the presence of anchor objects, we were able to show that both eye movements and body locomotion are strongly guided by these anchor objects when carrying out actions with naturalistic 3D settings. Overall, the data I will present will provide further evidence for the crucial role that anchor objects play in structuring the composition of scenes and thereby critically affecting visual search, object perception and the forming of memory representations in naturalistic environments.

(In what sense) We see social relations

Liuba Papeo^1,2; ¹CNRS, ²Université Claude Bernard Lyon

The most basic social relation is realized when two social agents engage in a physical exchange, or interaction. How do representations of social interactions come about, from basic processing in visual perception? Behavioral and neuroimaging phenomena show that human vision (and selective areas of the visual cortex) discriminates between scenes involving the same bodies, based on whether the individuals appear to interact or not. What information in a multiple-body scene channels the representation of social interaction? And what exactly is represented of a social relation in the visual system? I will present behavioral results, based on a switch cost paradigm, showing that the visual system exploits mere spatial information (i.e., relative position of bodies in space and posture features) to “decide” not only whether there is an interaction or not, but also who the agent and the patient are. Another set of results, based on a backward masking paradigm, shows that the visual processing of socially-relevant spatial relations is agnostic to the content of the interaction, and indeed segregated from, and prior to, (inter)action identification. Thus, drawing a divide between perception and cognition, the current results suggest that the visual representation of social relations corresponds to a configuration of parts (bodies/agents) that respect the spatial relations of a prototypical social interaction –a sort of social-template, theoretically analogous to the face- or body-template in the visual system– before inference. How specific/general to different instances of social interaction this template is will be the main focus of my discussion.

The role of part-whole relations in scene processing

Daniel Kaiser¹; ¹Justus-Liebig-Universität Gießen

Natural scenes are not arbitrary arrangements of unrelated pieces of information. Their composition rather follows statistical regularities, with meaningful information appearing in predictable ways across different parts of the scene. Here, I will discuss how characteristic relations across different scene parts shape scene processing in the visual system. I will present recent research, in which I used variations of a straightforward “jumbling” paradigm, whereby scenes are dissected into multiple parts that are then either re-assembled into typical configurations (preserving part-whole relations) or shuffled to appear in atypical configurations (disrupting part-whole relations). In a series of fMRI and EEG studies, we showed that the presence of typical part-whole relations has a profound impact on visual processing. These studies yielded three key insights: First, responses in scene-selective cortex are highly sensitive to spatial part-whole relations, and more so for upright than for inverted scenes. Second, the presence of typical part-whole structure facilitates the rapid emergence of scene category information in neural signals. Third, the part-whole structure of natural scenes supports the perception and neural processing of task-relevant objects embedded in the scene. Together, these results suggest a configural code for scene representation. I will discuss potential origins of this configural code and its role in efficient scene parsing during natural vision.

Two Approaches to Visual Relations: Deep Learning versus Structural Models

Hongjing Lu¹, Phil Kellman¹; ¹University of California, Los Angeles

Humans are remarkably adept at seeing in ways that go well beyond pattern classification. We represent bounded objects and their shapes from visual input, and also extract meaningful relations among object parts and among objects. It remains unclear what representations are deployed to achieve these feats of relation processing in vision. Can human perception of relations be best emulated by applying deep learning models to massive numbers of problems, or should learning instead focus on acquiring structural representations, coupled with the ability to compute similarities based on such representations? To address this question, we will present two modeling projects, one on abstract relations in shape perception, and one on visual analogy based on part-whole relations. In both projects we compare human performance to predictions derived from various deep learning models and from models based on structural representations. We argue that structural representations at an abstract level play an essential role in facilitating relation perception in vision.

< Back to 2022 Symposia

Beyond objects and features: High-level relations in visual perception

Friday, May 13, 2022, 12:00 – 2:00 pm EDT, Talk Room 1

Organizers: Chaz Firestone¹, Alon Hafri¹; ¹Johns Hopkins University

The world contains not only objects and features (red apples, glass bowls, large dogs, and small cats), but also relations holding between them (apples contained in bowls, dogs chasing cats). What role does visual processing play in extracting such relations, and how do relational representations structure visual experience? This symposium brings together a variety of approaches to explore new perspectives on the visual processing of relations. A unifying theme is that relations deserve equal place at the vision scientist’s table—and indeed that many traditional areas of vision science (including scene perception, attention, and memory) are fundamentally intertwined with relational representation. More…

Beyond representation and attention: Cognitive modulations of activity in visual cortex

Friday, May 13, 2022, 12:00 – 2:00 pm EDT, Talk Room 2

Organizers: Alex White¹, Kendrick Kay²; ¹Barnard College, Columbia University, ²University of Minnesota

This symposium addresses modulations of activity in visual cortex that go beyond classical notions of stimulus representation and attentional selection. For instance, activity patterns can reflect the contents of visual imagery, working memory, and expectations. In other cases, unstimulated regions of cortex are affected by the level of arousal or task difficulty. Furthermore, what might appear as general attentional amplifications are sometimes quite specific to stimulus type, brain region, and task. Although these effects are diverse, this symposium will seek unifying principles that are required to build general models of how sensory and cognitive signals are blended in visual cortex. More…

How we make saccades: selection, control, integration

Friday, May 13, 2022, 2:30 – 4:30 pm EDT, Talk Room 1

Organizers: Emma Stewart¹, Bianca R. Baltaretu¹; ¹Justus-Liebig University Giessen, Germany

Making a saccade is a non-trivial process: the saccade target must be selected, the visuomotor system must execute a motor command, and the visual system must integrate pre- and postsaccadic information. Recent research has uncovered titillating new roles for established neural regions, giving an evolving and sophisticated perspective into processes underlying saccadic selection and control. Additionally, computational models have advanced our understanding of how saccades shape perception. This symposium will unify established knowledge about the disparate phases of saccade production, giving insight into the full life cycle of a saccade, from selection, to control, to the ultimate ensuing transsaccadic perception. More…

Perceptual Organization – Lessons from Neurophysiology, Human Behavior, and Computational Modeling

Friday, May 13, 2022, 2:30 – 4:30 pm EDT, Talk Room 2

Organizers: Dirk B. Walther¹, James Elder²; ¹University of Toronto, ²York University

A principal challenge for both biological and machine vision systems is to integrate and organize the diversity of cues received from the environment into the coherent global representations we experience and require to make good decisions and take effective actions. Early psychological investigations date back more than 100 years to the seminal work of the Gestalt school. But in the last 50 years, neuroscientific and computational approaches to understanding perceptual organization have become equally important, and a full understanding requires integration of all three approaches. This symposium will highlight the latest results and identify promising directions in perceptual organization research. More…

The probabilistic nature of vision: How should we evaluate the empirical evidence?

Friday, May 13, 2022, 5:00 – 7:00 pm EDT, Talk Room 1

Organizers: Ömer Dağlar Tanrıkulu¹, Arni Kristjansson²; ¹Williams College, ²University of Iceland

The view that our visual system represents sensory information probabilistically is prevalent in contemporary vision science. However, providing empirical evidence for such a claim has proved to be difficult since both probabilistic and non-probabilistic perceptual representations can, in principle, account for the experimental results in the literature. In this symposium, we discuss how vision research can provide empirical evidence relevant to the question of probabilistic perception. How can we operationalize probabilistic visual representations, and, if possible, how can we provide empirical evidence that settles the issue? Our goal is to encourage researchers to make their assumptions about probabilistic perception explicit. More…

What does the world look like? How do we know?

Friday, May 13, 2022, 5:00 – 7:00 pm EDT, Talk Room 2

Organizers: Mark Lescroart¹, Benjamin Balas², Kamran Binaee¹, Michelle Greene³, Paul MacNeilage¹; ¹University of Nevada, Reno, ²North Dakota State University, ³Bates College

Statistical regularities in visual experience have been broadly shown to shape neural and perceptual visual processing. However, our ability to make inferences about visual processing based on natural image statistics is limited by the representativeness of natural image datasets. Here, we consider the consequences of using non-representative datasets, and we explore challenges in assembling datasets that are more representative in terms of the sampled environments, activities, and individuals. We explicitly address the following questions: what are we not sampling, why are we not sampling it, and how does this limit the inferences we can draw about visual processing? More…

Organizers: Oliver Braddick¹, Janette Atkinson²; ¹University of Oxford, ²University College London
Presenters: Oliver Braddick, Rebecca Saxe, T. Rowan Candy, Dennis M Levi, Janette Atkinson, Tessa Dekker

< Back to 2021 Symposia

In the last two decades, the science of human development has moved beyond defining how and when basic visual functions emerge during infancy and childhood, through both technical and conceptual advances. First, technical progress in MRI and near infrared spectroscopy and dedicated efforts by researchers have made it possible to image and localize activity in the visual brain, as early as the first months of life. These will be exemplified in the symposium by Rebecca Saxe’s presentation on the development of area specialization within the ventral visual stream in early infancy, and in Tessa Dekker’s research on childhood development of decision making for efficient visual cue combination, combining neuroimaging with novel behavioural measures .Secondly, Rowan Candy’s presentation will show how measurements of infants’ eye movements and refractive state using Purkinje image eye tracking and photorefraction have been refined to new levels of accuracy, providing novel insights into oculomotor development and how it interacts with binocular visual experience from the first weeks of life. This work offers the possibility of understanding the early development of strabismus where accommodation-vergence synergy develops atypically. The resulting condition of amblyopia reflects early plasticity, but the work presented by Dennis Levi shows that this condition remains treatable into adulthood, using novel therapies designed to re-establish binocular interactions rather than simply strengthen the cortical input from the amblyopic eye, with new implications for extended critical periods . Third, these approaches, alongside new behavioural methods , have highlighted the interlocking relationships between basic visual functions and visuocognitive processes such as decision-making and attention. Janette Atkinson’s presentation will define the key role of attention in visual development, and how different components of attention, depending on distinct brain networks, can be separated in young children and those with neurodevelopmental disorders. Different disorders (e.g. perinatal brain damage, Down and Williams syndromes) show distinctive profiles of attentional impairment. Imaging studies of cortical area and fibre tract development suggest that specific parietal and frontal networks are associated with individual differences in children’s visual decision-making and may also develop atypically across many developmental disorders. Tessa Dekker’s presentation will show how decision processes operating on visual information are as critical in development, including visuomotor development, as the development of basic sensitivity to visual feature properties. Detailed modelling of visual and visuomotor behaviour and localised brain responses indicate a prolonged development into middle & late childhood of the integrative processes required for efficient visual decisions. These talks illustrate some highlights in a much wider field of new insights into both typical and atypical visual development. Oliver Braddick’s presentation will outline the scope of this broader field, including pointers to work on automated visual assessment, infant eye-tracking with head-mounted cameras in the natural visual environment, isolating specific discriminations through frequency-tagging EEG, MRI analyses of developing brain connectivity, and the developmental impact of early and late visual deprivation. This whole range of work has greatly extended our understanding of the developing visual brain and its intimate links throughout neurocognitive systems, and allows us to identifiy the challenges ahead.

Presentations

New techniques, new questions in visual development

Oliver Braddick¹; ¹University of Oxford

In the last two decades, the range of research on visual development has been expanded by new methodologies, some represented in this symposium, which provide richer data and more direct insights into the visual brain mechanisms underlying development. This talk provides a brief overview of other advances which have started to answer some key questions in visual development: (i) application of eye tracking to automated visual assessment; (ii) head-mounted eye tracking yielding data on how infants sample their natural visual environment; (iii) frequency-tagging to refine the specificity of information yielded by EEG; (iv) MRI approaches to the connectivity and structure of the developing visual brain, including individual differences in development; (v) broader studies of the impact of visual deprivation on human visual development. As well as applying new methods, developmental research, in common with vision research more generally, has also extended its scope into the interfaces of vision with attention, action systems, decision processes, and other aspects of cognition. All these advances open the prospects of a wider and deeper understanding of the role of vision in the development of brain systems in infancy and childhood. However, there remain challenges in understanding the origins of individual differences across children in visuospatial, visuomotor, and visuosocial cognition.

The origins of specificity in ventral-stream cortical areas

Rebecca Saxe¹, Heather Kosakowski¹, Lindsey Powell¹, Michael Cohen¹; ¹Massachussetts Institute of Technology

In human adults, visual responses to object categories are organized into large scale maps; and within these maps are regions responding highly-selectively to behaviourally significant stimulus categories, such as faces, bodies, and scenes. Here we used fMRI in awake infants to directly measure the early development of these visual responses. Our first study (n=9) found that extrastriate cortex of 4–6-month-old infants contains regions that respond preferentially to faces, scenes, and objects with a spatial organization similar to adults. However, the responses in these regions were not selective for a single category. In a second fMRI study (n=26, age 2-9 months) we replicated these results, again finding preferential responses to faces, scenes and objects (but not bodies) in extrastriate areas. Third, we again replicated spatially separable responses to faces and scenes, but not bodies, within lateral occipito-temporal cortex, using functional near infrared spectroscopy (fNIRS). These results demonstrate that the large-scale organization of category preferences in visual cortex is present within a few months after birth, but is subsequently refined through development.

Infants’ control of their visual experience through vergence and accommodation

T. Rowan Candy¹; ¹University of Indiana

While a large literature has demonstrated the impact of abnormal visual experience on postnatal development of the visual system, the role of the ocular motor visual system in defining retinal visual experience during infancy and early childhood has been less well understood. Advances in instrumentation have made it possible for us to track simultaneously infants’ vergence eye movements and accommodation, showing that these responses are coupled, associated with sensitivity to binocular disparity, and can be dynamically adjusted, from the first weeks of life. This control, along with that of conjugate eye movements, enables infants to control their own visual experience in their dynamic three-dimensional world. In turn, visual experience enables most children to calibrate these coupled responses effectively, while others develop misalignment of their eyes and strabismus. A key question for future studies is to explore the source of this individual failure, whether it lies in disrupted fusional vergence potential or in the ability to undergo adaptation. This talk will also briefly consider the following questions: How does the improving spatial resolution of the infant’s visual system affect the iterative development of motor and sensory visual systems? How can human visual development inform machine learning and robotics? How does development of the first stages of visual processing impact higher-order extrastriate function, and what is the influence of top-down processes?

Rethinking amblyopia and its therapies

Dennis M Levi¹; ¹University of California, Berkeley

Recent work has transformed our ideas about effective therapies for amblyopia. Since the 1700’s, the clinical treatment for amblyopia has consisted of patching or penalizing the strong eye, to force the “lazy” amblyopic eye, to work. This treatment has generally been limited to infants and young children during the “critical” or sensitive period of development. Over the last 20 years, we have learned much about the nature and neural mechanisms underlying the loss of spatial and binocular vision in amblyopia, and that a degree of neural plasticity persists well beyond the sensitive period. Importantly, the last decade has seen a resurgence of research into new approaches to the treatment of amblyopia both in children and adults, which emphasise that monocular therapies may not be the most effective for the fundamentally binocular disorder that is amblyopia. These approaches include perceptual learning, video game play and binocular methods aimed at reducing inhibition of the amblyopic eye by the strong fellow eye, and enhancing binocular fusion and stereopsis. This talk will highlight both the successes of these approaches in labs around the world, and their dismal failures in clinical trials. Reconciling these results raises important new questions that may help to focus future directions.

Typical and atypical brain development for components of visual attention

Janette Atkinson¹; ¹University College London

The development of attention mechanisms plays a key role in how visual information is used and in determining how the visual environment shapes visual development. However, visual attention is not a unitary process but involves multiple components of selective attention, sustained attention, and executive function(s). The Early Childhood Attention Battery (ECAB) allows these components to be separated in preschool children or children with equivalent mental age and defines individual and group differences in their ‘attention profile’ across these components. For example, we find that sustained visual attention is impaired in children with perinatal brain injury and/or very premature birth, but that this ability is relatively preserved in children with Williams (WS) and Down Syndrome (DS). Children with DS or WS have difficulties inhibiting prepotent responses in executive function tasks, although in WS these difficulties are much greater in the visuospatial than the vebal domain. Spatial attention processes are particularly associated with structures in the dorsal stream and our past work has highlighted attention as an element of the ‘dorsal stream vulnerability’ characterising many developmental disorders. We will discuss these patterns of deficit across syndromes in relation to the dorsal and ventral attention networks and salience network defined by data from current connectivity studies in children and adults, including our findings on the tracts associated with children’s performance on visual decisions. Individual variations in the way these networks interact may determine the way top-down goals and bottom-up sensory stimulation are integrated in the control of visual behaviour in development.

Model-based MRI and psychophysics reveal crucial role of decision-making in visual development in childhood.

Tessa Dekker¹, Marko Nardini², Peter Jones¹; ¹University College London, ²University of Durham

Vision undergoes major development during infancy and childhood, demonstrated in improvements in both detection and recognition tasks. Classically, developmental vision research has focussed on sensitivity improvements in early visual channels. However, in recent years, decision-theoretic approaches have formalised how changes in visual performance could also result from more efficient use of available information, for example by optimising decision rules, cost functions, and priors. Using these quantitative frameworks, we are beginning to understand how these factors contribute to childhood vision. For example, improved depth perception in late childhood reflects a shift from processing depth cues independently to combining them in visual cortex, as demonstrated by the emergence of fMRI evidence for fused depth-cue representations within neural detectors in area V3B. Similarly, development of visual motion-, location-, and object perception, in part reflects more efficient combining of stimulus features (e.g., averaging dots across displays) besides greater sensitivity to these features’ properties (e.g., single dot motion). Thus, rather than greater sensitivity to basic visual information, substantial improvements in visual discrimination and detection may reflect better inferential capacities. This also applies to visually-guided movement tasks that emulate real-life action under risk: while adults can rapidly identify visuomotor strategies that minimise risk and uncertainty in new situations with complex cost factors, children up to age 10 years do not. Together, these studies show that improved decision-making plays a major role in visual development in childhood, and that modelling this role is needed to gain computational-level insight in the driving factors of human visual plasticity.

< Back to 2021 Symposia

Organizers: Nir Shalev^1,2,3, Anna Christina (Kia) Nobre^1,2,3; ¹Department of Experimental Psychology, University of Oxford, ²Wellcome Centre for Integrative Neuroscience, University of Oxford, ³Oxford Centre for Human Brain Activity, University of Oxford
Presenters: Jennifer Coull, Rachel Denison, Shlomit Yuval-Greenberg, Nir Shalev, Assaf Breska, Sander Los

< Back to 2021 Symposia

The study of temporal preparation in guiding behaviour adaptively and proactively has long roots, traceable at least as far back as Wundt (1887). Additional forays into exploring the temporal dimension of anticipatory attention resurfaced during the early years of cognitive psychology. But, the field of selective temporal attention has undoubtedly blossomed in the last twenty years. In 1998, Coull and Nobre introduced a temporal analogue of the visual spatial orienting paradigm (Posner, 1980), demonstrating sizeable and reproducible effects of temporal orienting, as well as ushering in studies to study its neural systems and mechanisms. The studies built on seminal psychological demonstrations of auditory perceptual facilitation by temporal rhythms (Jones, 1976). Over the ensuing years, investigating ‘when we attend’ has become increasingly mainstay. Today we recognise that our psychological and neural systems extract temporal information from recurring temporal rhythms, associations, probabilities, and sequences to enhance perception in the various modalities as well as across them. Sophisticated experimental designs have been developed, and various approaches have been applied to investigate the principles of selective temporal attention. Are there dedicated systems for anticipating events in time leading to a common set of modulatory functions? Or, are mechanisms for temporal orienting embedded within task-specific systems and dependent on the nature of the available temporal regularities (e.g., rhythms or associations). In the following symposium, we illustrate contemporary research on selective temporal attention by bringing together researchers from across the globe and using complementary approaches. Across the presentations, researchers explore the roles of temporal rhythms, associations, probabilities, and sequences using psychophysics, eye movements, neural measurements, neuropsychology, developmental psychology, and theoretical models. In a brief introduction, Coull and Nobre will comment on the context of their initial temporal orienting studies and on the major strands and developments in the field. The first research presentation by Rachel Denison (with Marisa Carrasco) will introduce behavioural and neurophysiological studies demonstrating the selective nature of temporal attention and its relative costs and benefits to performance. The second presentation by Shlomit Yuval-Greenberg will show how anticipatory temporal attention influences oculomotor behaviour, with converging evidence from saccades, micro-saccades, and eye-blinks. The third presentation by Nir Shalev (with Sage Boettcher) will show how selective temporal attention generalises to dynamic and extended visual search contexts, picking up on learned conditional probabilities to guide perception and eye movement in adults and in children. The fourth presentation by Assaf Breska will provide evidence for a double dissociation between temporal attention based on temporal rhythms vs. associations by comparing performance of individuals with lesions in the cerebellum vs. basal ganglia. The final presentation by Sander Los will introduce a theoretical and computational model that proposes to account to various effects of temporal orienting across multiple time spans – from between successive trials to across contexts. A panel discussion will follow, to consider present and forthcoming research challenges and opportunities. In addition to considering current issues in selective temporal attention, our aim is to lure our static colleagues into the temporal dimension.

Presentations

20 years of temporal orienting: an introduction

Jennifer Coull^1,2, Anna Christina Nobre^3,4,5; ¹Aix-Marseille Universite, France, ²French National Center for Scientific Research (CNRS), ³Department of Experimental Psychology, University of Oxford, ⁴Wellcome Centre for Integrative Neuroscience, University of Oxford, ⁵Oxford Centre for Human Brain Activity, University of Oxford

In a brief introduction to the symposium, we will spell out the main questions and issues framing cognitive neuroscience studies of attention when we conducted our first temporal orienting combining behavioural methods with PET, fMRI, and ERPS. We will reflect on the strands of research at the time which helped guide our thinking and interpretation of results; and then consider the rich, varied, and many ways in which the temporal attention field has evolved into its exciting, dynamic, and multifaceted guise.

The dynamics of temporal attention

Rachel Denison¹, Marisa Carrasco¹; ¹Department of Psychology, New York University

Selection is the hallmark of attention: processing improves for attended items but is relatively impaired for unattended items. It is well known that visual spatial attention changes sensory signals and perception in this selective fashion. In the research we will present, we asked whether and how attentional selection happens across time. Specifically, we investigated voluntary temporal attention, the goal-driven prioritization of visual information at specific points in time. First, our experiments revealed that voluntary temporal attention is selective, resulting in perceptual tradeoffs across time. Perceptual sensitivity increased at attended times and decreased at unattended times, relative to a neutral condition in which observers were instructed to sustain attention. Temporal attention changed the precision of orientation estimates, as opposed to an all-or-none process, and it was similarly effective at different visual field locations (fovea, horizontal meridian, vertical meridian). Second, we measured microsaccades and found that directing voluntary temporal attention increases the stability of the eyes in anticipation of a brief, attended stimulus, improving perception. Attention affected microsaccade dynamics even for perfectly predictable stimuli. Precisely timed gaze stabilization can therefore be an overt correlate of the allocation of temporal attention. Third, we developed a computational model of dynamic attention, which incorporates normalization and dynamic gain control, and accounts for the time-course of perceptual tradeoffs. Altogether, this research shows how voluntary temporal attention increases perceptual sensitivity at behaviorally relevant times, and helps manage inherent limits in visual processing across short time intervals. This research advances our understanding of attention as a dynamic process.

Oculomotor inhibition as a correlate of temporal orienting

Shlomit Yuval-Greenberg^1,2, Noam Tal¹, Dekel Abeles¹; ¹School of Psychological Sciences, Tel-Aviv University, ²Sagol School of Neuroscience, Tel-Aviv University

Temporal orienting in humans is typically assessed by measuring classical behavioral measurements, such as reaction times (RTs) and accuracy-rates, and by examining electrophysiological responses. But these methods have some disadvantages: RTs and accuracy-rates provide only retrospective estimates of temporal orientation, and electrophysiological markers are often difficult to interpret. Fixational eye movements, such as microsaccades, occur continuously and involuntarily even when observers attempt to suppress them by holding steady fixation. These continuous eye movements can provide reliable and interpretable information on fluctuations of cognitive states across time, including those that are related to temporal orienting. In a series of studies, we show that temporal orienting is associated with the inhibition of oculomotor behaviors, including saccades, microsaccades and eye-blinks. First, we show that eye movements are inhibited prior to predictable visual targets. This effect was found for targets that were anticipated either because they were embedded in a rhythmic stream of stimulation or because they were preceded by an informative temporal cue. Second, we show that this effect is not specific to the visual modality but is present also for temporal orienting in the auditory modality. Last, we show that the oculomotor inhibition effect of temporal orienting is related to the construction of expectations and not to the estimation of interval duration, and also that it reflects a local trial-by-trial anticipation rather than a global arousal state. We conclude that pre-target inhibition of oculomotor behaviors is a reliable correlate of temporal orienting processes of various types and modalities.

Spatial-temporal predictions in a dynamic visual search

Nir Shalev^1,2,3, Sage Boettcher^1,2,3, Anna Christina Nobre^1,2,3; ¹Department of Experimental Psychology, University of Oxford, ²Wellcome Centre for Integrative Neuroscience, University of Oxford, ³Oxford Centre for Human Brain Activity, University of Oxford

Our environment contains many regularities that allow the anticipation of upcoming events. Waiting for a traffic light to change, an elevator to arrive, or using a toaster: all contain temporal ‘rules’ that can be learned and used to improve performance. We investigated the guidance of spatial attention based on spatial-temporal associations using a dynamic variation of a visual search task. On each trial, individuals searched for eight targets among distractors, all fading in and out of the display at different locations and times. The screen was split into four distinct quadrants. Crucially, we rendered four targets predictable by presenting them repeatedly in the same quadrants and times throughout the task. The other four targets were randomly distributed in their locations and onsets. At the first part of our talk, we will show that participants are faster and more accurate in detecting predictable targets. We identify this benefit when testing both young adults (age 18-30), and in a cohort of young children (age 5-6). At the second part of the talk, we will present a further inquiry about the source of the behavioural benefit, contrasting sequential-priming vs. memory guidance. We do so by introducing two more task variations: one in which the onsets and locations of all targets occasionally repeated in successive trials; and one in which the trial pattern was occasionally violated. The results suggest that both factors, i.e., priming and memory, provide a useful source for guiding attention.

Distinct mechanisms of rhythm- and interval-based attention shifting in time

Assaf Breska¹; ¹Department of Psychology, University of California, Berkeley, ²Helen Wills Neuroscience Institute, University of California, Berkeley

A fundamental principle of brain function is the use of temporal regularities to predict the timing of upcoming events and proactively allocate attention in time accordingly. Historically, predictions in rhythmic streams were explained by oscillatory entrainment models, whereas predictions formed based on associations between cues and isolated interval were explained by dedicated interval timing mechanisms. A fundamental question is whether predictions in these two contexts are indeed mediated by distinct mechanisms, or whether both rely on a single mechanism. I will present a series of studies that combined behavioral, electrophysiological, neuropsychological and computational approached to investigate the cognitive and neural architecture of rhythm- and interval-based predictions. I will first show that temporal predictions in both contexts similarly modulate behavior and anticipatory neural dynamics measured by EEG such as ramping activity, as well as phase-locking of delta-band activity, previously taken as signature of oscillatory entrainment. Second, I will show that cerebellar degeneration patients were impaired in forming temporal predictions based on isolated intervals but not based on rhythms, while Parkinson’s disease patients showed the reverse pattern. Finally, I will demonstrate that cerebellar degeneration patients show impaired temporal adjustment of ramping activity and delta-band phase-locking, as well as timed suppression of beta-band activity during interval-based prediction. Using computational modelling, I will identify the aspects of neural dynamics that prevail in rhythm-based prediction despite impaired interval-based prediction. To conclude, I will discuss implications for rhythmic entrainment and interval timing models, and the role of subcortical structures in temporal prediction and attention.

Is temporal orienting a voluntary and controlled process?

Sander Los¹, Martijn Meeter¹, Wouter Kruijne²; ¹Vrije Universiteit Amsterdam, ²University of Groningen

Temporal orienting involves the allocation of attentional resources to future points in time to facilitate the processing of an expected target stimulus. To examine temporal orienting, studies have varied the foreperiod between a warning stimulus and a target stimulus, with a cue specifying the duration of the foreperiod at the start of each trial with high validity (typically 80%). It has invariably been found that the validity of the cue has a substantial behavioral effect (typically expressed in reaction times) on short-foreperiod trials but not on long-foreperiod trials. The standard explanation of this asymmetry starts with the idea that, at the start of each trial, the participant voluntarily aligns the focus of attention with the moment specified by the cue. On short foreperiod trials, this policy leads to an effect of cue validity, reflecting differential temporal orienting. By contrast, on long-foreperiod trials, an initially incorrect early focus of attention (induced by an invalid cue) will be discovered during the ongoing foreperiod, allowing re-orienting toward a later point in time, thus preventing behavioral costs. In this presentation, we challenge this view. Starting from our recent multiple trace theory of temporal preparation (MTP), we developed an alternative explanation based on the formation of associations between the specific cues and foreperiods. We will show that MTP accounts naturally for the typical findings in temporal orienting without recourse to voluntary and controlled processes. We will discuss initial data that serve to distinguish between the standard view and the view derived from MTP.

< Back to 2021 Symposia

Organizers: Susan Wardle¹, Chris Baker¹; ¹National Institutes of Health
Presenters: Aina Puce, Frank Tong, Janneke Jehee, Justin Gardner, Marieke Mur

< Back to 2021 Symposia

Over the past 20 years, neuroimaging methods have become increasingly popular for studying the neural mechanisms of vision in the human brain. To celebrate 20 years of VSS this symposium will focus on the contribution that brain imaging techniques have made to our field of vision science. In the year 2000, we knew about retinotopy and category-selectivity, but neuroimaging was still evolving. Now in 2020, the field is taking an increasingly computational approach to applying neuroimaging data to understanding questions about vision. The aim of this symposium is to provide both a historical context and a forward-focus for the role of neuroimaging in vision science. Our speakers are a diverse mix of pioneering researchers in the field who applied neuroimaging in the early days of the technique, and those who have more recently continued to push the field forward by creative application of imaging techniques. We have also selected speakers who use a range of different methodological approaches to investigate both low-level and high-level vision, including computational and modeling techniques, multivariate pattern analysis and representational similarity analysis, and methods that aim to link brain to behavior. The session will begin with a short 5-10 min Introductory talk by Susan Wardle to provide context for the symposium. Talks by the five selected speakers will be 20 minutes each; with 1-2 mins available for clarification questions after each talk. The session will end with a longer 10-15 min general discussion period. In the first talk, Aina Puce will consider the contribution made by multiple neuroimaging techniques such as fMRI and M/EEG towards understanding the social neuroscience of face perception, and how technological advances are continuing to shape the field. In the second talk, Frank Tong will discuss progress made in understanding top-down feedback in the visual system using neuroimaging, predictive coding models, and deep learning networks. In the third talk, Janneke Jehee will argue that a crucial next step in visual neuroimaging is to connect cortical activity to behavior, using perceptual decision-making as an illustrative example. In the fourth talk, Justin Gardner will discuss progress made in using neuroimaging to link cortical activity to human visual perception, with a focus on quantitative linking models. In the final talk, Marieke Mur will reflect on what fMRI has taught us about high-level visual processes, and outline how understanding the temporal dynamics of object recognition will play an important role in the development of the next generation of computational models of human vision. Overall, the combination of a historical perspective and an overview of current trends in neuroimaging presented in this symposium will lead to informed discussion about what future directions will prove most fruitful for answering fundamental questions in vision science.

Presentations

Technological advances are the scaffold for propelling science forward in social neuroscience

Aina Puce¹; ¹Indiana University

Over the last 20 years, neuroimaging techniques [e.g. EEG/MEG, fMRI] were used to map neural activity within a core and extended brain network to study how we use social information from faces. By the 20th century’s end, neuroimaging methods had identified the building blocks of this network, but how these parts came together to make a whole was unknown. In 20 years, technological advances in data acquisition and analysis have occurred in a number of spheres. First, network neuroscience has progressed our understanding of which brain regions functionally connect with one another on a regular basis. Second, improvements in white matter tract tracing have allowed putative underlying white matter pathways to be identified for some functional networks. Third, [non-]invasive brain stimulation has allowed the identification of some causal relationships between brain activity and behavior. Fourth, technological developments in portable EEG and MEG systems propelled social neuroscience out of the laboratory and into the [ecologically valid] wide world. This is changing activation task design as well as data analysis. Potential advantages of these ‘wild type’ approaches include the increased signal-to-noise provided by a live interactive 3D visual stimulus e.g. another human being, instead of an isolated static face on a computer monitor. Fifth, work with machine learning algorithms has begun to differentiate brain/non-brain activity in these datasets. Finally, we are finally ‘putting the brain back into the body’ – whereby recordings of brain activity are made in conjunction with physiological signals including EKG, EMG, pupil dilation, and eye position.

Understanding the functional roles of top-down feedback in the visual system

Frank Tong¹; ¹Vanderbilt University

Over the last 20 years, neuroimaging techniques have shed light on the modulatory nature of top-down feedback signals in the visual system. What is the functional role of top-down feedback and might there be multiple types of feedback that can be implemented through automatic and controlled processes? Studies of voluntary covert attention have demonstrated the flexible nature of attentional templates, which can be tuned to particular spatial locations, visual features or to the structure of more complex objects. Although top-down feedback is typically attributed to visual attention, there is growing evidence that multiple forms of feedback exist. Studies of visual imagery and working memory indicate the flexible nature of top-down feedback from frontal-parietal areas to early visual areas for maintaining and manipulating visual information about stimuli that are no longer in view. Theories of predictive coding propose that higher visual areas encode feedforward signals according to learned higher order patterns, and that any unexplained components are fed back as residual error signals to lower visual areas for further processing. These feedback error signals may serve to define an image region as more salient, figural, or stronger in apparent contrast. Here, I will discuss both theory and supporting evidence of multiple forms of top-down feedback, and consider how deep learning networks can be used to evaluate the utility of predictive coding models for understanding vision. I will go on to discuss what important questions remain to be addressed regarding the nature of feedback in the visual system.

Using neuroimaging to better understand behavior

Janneke Jehee^1,2; ¹Donders Institute for Brain, Cognition and Behavior, ²Radboud University Nijmegen, Nijmegen, Netherlands

Over the past 20 years, functional MRI has become an important tool in the methodological arsenal of the vision scientist. The technique has led to many amazing discoveries, ranging from human brain areas involved in face perception to information about stimulus orientation in early visual activity. While providing invaluable insights, most of the work to date has sought to link visual stimuli to a cortical response, with far less attention paid to how such cortical stimulus representations might give rise to behavior. I will argue that a crucial next step in visual neuroimaging is to connect cortical activity to behavior, and will illustrate this using our recent work on perceptual decision-making.

Using neuroimaging to link cortical activity to human visual perception

Justin Gardner¹; ¹Stanford University

Over the last 20 years, human neuroimaging, in particular BOLD imaging, has become the dominant technique for determining visual field representations and measuring selectivity to various visual stimuli in the human cortex. Indeed, BOLD imaging has proven decisive in settling long standing disputes that other techniques such as electrophysiological recordings of single neurons provided only equivocal evidence for. For example, by showing that cognitive influences due to attention or perceptual state could be readily measured in so-called early sensory areas. Part of this success is due to the ability to make precise behavioral measurements through psychophysics in humans which can quantitatively measure such cognitive effects. Leveraging this ability to make quantitive behavioral measurements with concurrent measurement of cortical activity with BOLD imaging, we can provide answers to a central question of visual neuroscience: What is the link between cortical activity and perceptual behavior? To make continued progress in the next 20 years towards answering this question, we must turn to quantitative linking models that formalize hypothesized relationships between cortical activity and perceptual behavior. Such quantitative linking models are falsifiable hypotheses whose success or failure can be determined by their ability or inability to quantitatively account for behavioral and neuroimaging measurements. These linking models will allow us to determine the cortical mechanisms that underly visual perception and account for cognitive influences such as attention on perceptual behavior.

High-level vision: from category selectivity to representational geometry

Marieke Mur¹; ¹Western University, London ON, Canada

Over the last two decades, functional magnetic resonance imaging (fMRI) has provided important insights into the organization and function of the human visual system. In this talk, I will reflect on what fMRI has taught us about high-level visual processes, with an emphasis on object recognition. The discovery of object-selective and category-selective regions in high-level visual cortex suggested that the visual system contains functional modules specialized for processing behaviourally relevant object categories. Subsequent studies, however, showed that distributed patterns of activity across high-level visual cortex also contain category information. These findings challenged the idea of category-selective modules, suggesting that these regions may instead be clusters in a continuous feature map. Consistent with this organizational framework, object representations in high-level visual cortex are at once categorical and continuous: the representational code emphasizes category divisions of longstanding evolutionary relevance while still distinguishing individual images. This body of work provides important insights on the nature of high-level visual representations, but it leaves open how these representations are dynamically computed from images. In recent years, deep neural networks have begun to provide a computationally explicit account of how the ventral visual stream may transform images into meaningful representations. I will close off with a discussion on how neuroimaging data can benefit the development of the next generation of computational models of human vision and how understanding the temporal dynamics of object recognition will play an important role in this endeavor.

< Back to 2021 Symposia

Organizers: Rich Krauzlis¹, Michele Basso²; ¹National Eye Institute, ²Brain Research Institute, UCLA
Presenters: Ziad Hafed, Farran Briggs, Jude Mitchell, Marlene Cohen, Yasmine El-Shamayleh, Bevil Conway

< Back to 2021 Symposia

The symposium includes six highly regarded mid-career and junior investigators (Ziad Hafed, Farran Briggs, Jude Mitchell, Marlene Cohen, Yasmine El-Shamayleh, Bevil Conway) who use NHPs to study a range of topics (e.g., attention, eye movements, object and color perception) of interest to the VSS membership. Each speaker will have 15’ for their talk plus time for questions. Ziad Hafed will review how an observation about fixational eye movements, first described about 20 years ago, led to a series of studies exploring the brain circuits that control both attention and saccades. We now have a much deeper understanding of the underlying roots and implications of the correlation between attention and microsaccades, the role of subcortical and cortical visual structures in this process, and the importance of approaching vision from an active, rather than passive, perspective. Farran Briggs will describe a series of approaches using traditional and modern tools to explore how cortical feedback influences early visual processing. Transformations in visual signals traversing the feedforward retino-geniculo-cortical pathways are well understood, but the contribution of corticogeniculate feedback to visual perception is less clear. Through examinations of the morphology, physiology and function of corticogeniculate neurons, a new hypothesis emerges in which corticogeniculate feedback regulates the timing and precision of feedforward visual signal transmission. Jude Mitchell will describe the role of different classes of neurons in visual cortex. Over the past twenty years, there have been major advances towards manipulating and tagging different neuronal classes, and new molecular and recording techniques that distinguish cell class are now becoming available for use in NHPs. Jude will describe the application of these approaches in the marmoset monkey to understand how eye movements modulate early sensory processing as a function of cell class and cortical layer. Marlene Cohen will describe insights in understanding populations of neurons. Twenty years ago, most NHP work focused on the activity of single neurons and relatively simple stimuli and behaviors. It is now possible to record from many neurons in multiple brain areas while monkeys make judgments about a variety of stimulus properties. Marlene will describe recent work showing that these complex data sets can reveal strikingly simple relationships between neuronal populations and visual perception. Yasmine El-Shamayleh will review a well-established framework for studying visual object processing in the primate cerebral cortex – one that has stood the test of two decades of experimental investigation. She will then describe how she plans to leverage new, cell type-specific viral vector-based optogenetic approaches to begin to elucidate the detailed circuit-level mechanisms in extrastriate cortex that govern this visual function. Bevil Conway will discuss how functional MRI in NHPs has advanced our understanding of the ventral visual pathway. Combining fMRI with neurophysiology has facilitated the systematic study of extrastriate cortex, guided targeted recordings from neurons in functionally identified patches of cortex, and provided a direct comparison of brain activity in humans and monkeys. This work underscores the importance of understanding how functionally identified populations of neurons interact to enable perception of colors, objects, places and faces.

Presentations

Foveal action for the control of extrafoveal vision

Ziad Hafed¹; ¹Eberhard Karls Universität Tübingen

Ziad Hafed will review how an observation about fixational eye movements, first described about 20 years ago, led to a series of studies exploring the brain circuits that control both attention and saccades. We now have a much deeper understanding of the underlying roots and implications of the correlation between attention and microsaccades, the role of subcortical and cortical visual structures in this process, and the importance of approaching vision from an active, rather than passive, perspective.

The role of corticogeniculate feedback in visual perception

Farran Briggs¹; ¹University of Rochester

Farran Briggs will describe a series of approaches using traditional and modern tools to explore how cortical feedback influences early visual processing. Transformations in visual signals traversing the feedforward retino-geniculo-cortical pathways are well understood, but the contribution of corticogeniculate feedback to visual perception is less clear. Through examinations of the morphology, physiology and function of corticogeniculate neurons, a new hypothesis emerges in which corticogeniculate feedback regulates the timing and precision of feedforward visual signal transmission.

Neural circuits for pre-saccadic attention in the marmoset monkey

Jude Mitchell¹; ¹University of Rochester

Jude Mitchell will describe the role of different classes of neurons in visual cortex. Over the past twenty years, there have been major advances towards manipulating and tagging different neuronal classes, and new molecular and recording techniques that distinguish cell class are now becoming available for use in NHPs. Jude will describe the application of these approaches in the marmoset monkey to understand how eye movements modulate early sensory processing as a function of cell class and cortical layer.

Multi-neuron approaches to studying visual perception and decision-making

Marlene Cohen¹; ¹University of Pittsburgh

Marlene Cohen will describe insights in understanding populations of neurons. Twenty years ago, most NHP work focused on the activity of single neurons and relatively simple stimuli and behaviors. It is now possible to record from many neurons in multiple brain areas while monkeys make judgments about a variety of stimulus properties. Marlene will describe recent work showing that these complex data sets can reveal strikingly simple relationships between neuronal populations and visual perception.

Neural circuits for visual object processing

Yasmine El-Shamayleh¹; ¹Columbia University

Yasmine El-Shamayleh will review a well-established framework for studying visual object processing in the primate cerebral cortex – one that has stood the test of two decades of experimental investigation. She will then describe how she plans to leverage new, cell type-specific viral vector-based optogenetic approaches to begin to elucidate the detailed circuit-level mechanisms in extrastriate cortex that govern this visual function.

Parallel multi-stage processing of inferior temporal cortex: faces, objects, colors and places

Bevil Conway¹; ¹National Eye Institute

Bevil Conway will discuss how functional MRI in NHPs has advanced our understanding of the ventral visual pathway. Combining fMRI with neurophysiology has facilitated the systematic study of extrastriate cortex, guided targeted recordings from neurons in functionally identified patches of cortex, and provided a direct comparison of brain activity in humans and monkeys. This work underscores the importance of understanding how functionally identified populations of neurons interact to enable perception of colors, objects, places and faces.

< Back to 2021 Symposia

Organizers: Lisa Ostrin¹, David Brainard², Lynne Kiorpes³; ¹University of Houston College of Optometry, ²University of Pennsylvania, ³New York University
Presenters: Susana Marcos, Brian Vohnsen, Ann Elsner, Juliette E. McGregor

< Back to 2021 Symposia

This year’s biennial ARVO at VSS symposium focuses on early stages of visual processing at the fovea. Speakers will present recent work related to optical, vascular, and neural factors contributing to vision, as assessed with advanced imaging techniques. The work presented in this session encompasses clinical and translational research topics, and speakers will discuss normal and diseased conditions.

Presentations

Foveal aberrations and the impact on vision

Susana Marcos¹; ¹Institute of Optics, CSIC

Optical aberrations degrade the quality of images projected on the retina. The magnitude and orientation of the optical aberrations vary dramatically across individuals. Changes also occur with processes such as accommodation, and aging, and also with corneal and lens disease and surgery. Certain corrections such as multifocal lenses for presbyopia modify the aberration pattern to create simultaneous vision or extended depth-of-focus. Ocular aberrometers have made their way into the clinical practice. Besides, quantitative 3-D anterior segment imaging has allowed quantifying the morphology and alignment of the cornea and lens, linking ocular geometry and aberrations through custom eye models, and shedding light on the factors contributing to the optical degradation. However, perceived vision is affected by the eye’s aberrations in more ways than those purely predicted by optics, as the eye appears to be adapted to the magnitude and orientation of its own optical blur. Studies using Adaptive Optics, not only reveal the impact of manipulating the optical aberrations on vision, but also that the neural code for blur is driven by subject’s own aberrations.

The integrated Stiles-Crawford effect: understanding the role of pupil size and outer-segment length in foveal vision

Brian Vohnsen¹; ¹Advanced Optical Imaging Group, School of Physics, University College Dublin, Ireland

The Stiles-Crawford effect of the first kind (SCE-I) describes a psychophysical change in perceived brightness related to the angle of incidence of a ray of light onto the retina. The effect is commonly explained as being due to angular-dependent waveguiding by foveal cones, yet the SCE-I is largely absent from similar-shaped rods suggesting that a different mechanism than waveguiding is at play. To examine this, we have devised a flickering pupil method that directly measures the integrated SCE-I for normal pupil sizes in normal vision rather than relying on mathematical integration of the standard SCE-I function as determined with Maxwellian light. Our results show that the measured effective visibility for normal foveal vision is related to visual pigment density in the three-dimensional retina rather than waveguiding. We confirm the experimental findings with a numerical absorption model using Beer-Lambert’s law for the visual pigments.

Structure of cones and microvasculature in healthy and diseased eyes

Ann Elsner¹; ¹Indiana University School of Optometry

There are large differences in the distribution of cones in the living human retina, with the density at the fovea varying more than with greater eccentricities. The size and shape of the foveal avascular zone also varies across individuals, and distances between capillaries can be greatly enlarged in disease. While diseases such as age-related macular degeneration and diabetes impact greatly on both cones and retinal vessels, some cones can survive for decades although their distributions become more irregular. Surprisingly, in some diseased eyes, cone density at retinal locations outside those most compromised can exceed cone density for control subjects.

Imaging of calcium indicators in retinal ganglion cells for understanding foveal function

Juliette E. McGregor¹; ¹Centre for Visual Science, University of Rochester

The fovea mediates much of our conscious visual perception but is a delicate retinal structure that is difficult to investigate physiologically using traditional approaches. By expressing the calcium indicator protein GCaMP6s in retinal ganglion cells (RGCs) of the living primate we can optically read out foveal RGC activity in response to visual stimuli presented to the intact eye. Pairing this with adaptive optics ophthalmoscopy it is possible to both present highly stabilized visual stimuli to the fovea and read out retinal activity on a cellular scale in the living animal. This approach has allowed us to map the functional architecture of the fovea at the retinal level and to classify RGCs in vivo based on their responses to chromatic stimuli. Recently we have used this platform as a pre-clinical testbed to demonstrate successful restoration of foveal RGC responses following optogenetic therapy.

< Back to 2021 Symposia

Organizers: Shaul Hochstein¹, Merav Ahissar²; ¹Life Sciences, Hebrew University, Jerusalem, ²Psychology, Hebrew University, Jerusalem
Presenters: Jeremy M Wolfe, Shaul Hochstein, Catherine Tallon-Baudry, James DiCarlo, Merav Ahissar

< Back to 2021 Symposia

Forty years ago, Anne Treisman presented Feature Integration Theory (FIT; Treisman & Gelade, 1980). FIT proposed a parallel, preattentive first stage and a serial second stage controlled by visual selective attention, so that search tasks could be divided into those performed by the first stage, in parallel, and those requiring serial processing and further “binding” in an object file (Kahneman, Treisman, & Gibbs, 1992). Ten years later, Jeremy Wolfe expanded FIT with Guided Search Theory (GST), suggesting that information from the first stage could guide selective attention in the second (Wolfe, Cave & Franzel, 1989; Wolfe, 1994). His lab’s recent visual search studies enhanced this theory (Wolfe, 2007), including studies of factors governing search (Wolfe & Horowitz, 2017), hybrid search (Wolfe, 2012; Nordfang, Wolfe, 2018), and scene comprehension capacity (Wick … Wolfe, 2019). Another ten years later, Shaul Hochstein and Merav Ahissar proposed Reverse Hierarchy Theory (RHT; Hochstein, Ahissar, 2002), turning FIT on its head, suggesting that early conscious gist perception, like early generalized perceptual learning (Ahissar, Hochstein, 1997, 2004), reflects high cortical level representations. Later feedback, returning to lower levels, allows for conscious perception of scene details, already represented in earlier areas. Feedback also enables detail-specific learning. Follow up found that top-level gist perception primacy leads to the counter-intuitive results that faces pop out of heterogeneous object displays (Hershler, Hochstein, 2005), individuals with neglect syndrome are better at global tasks (Pavlovskaya … Hochstein, 2015), and gist perception includes ensemble statistics (Khayat, Hochstein, 2018, 2019; Hochstein et al., 2018). Ahissar’s lab mapped RHT dynamics to auditory systems (Ahissar, 2007; Ahissar etal., 2008) in both perception and successful/failed (from developmental disabilities) skill acquisition (Lieder … Ahissar, 2019) James DiCarlo has been pivotal in confronting feedforward-only versus recurrency-integrating network models of extra-striate cortex, considering animal/human behavior (DiCarlo, Zoccolan, Rust, 2012; Yarmins … DiCarlo, 2014; Yamins, DiCarlo, 2016). His large-scale electrophysiology recordings from behaving primate ventral stream, presented with challenging object-recognition tasks, relate directly to whether recurrent connections are critical or superfluous (Kar … DiCarlo, 2019). He recently developed combined deep artificial neural network modeling, synthesized image presentation, and electrophysiological recording to control neural activity of specific neurons and circuits (Bashivan, Kar, DiCarlo, 2019). Cathrine Tallon-Baudry uses MEG/EEG recordings to study neural correlates of conscious perception (Tallon-Baudry, 2012). She studied roles of human brain oscillatory activity in object representation and visual search tasks (Tallon-Baudry, 2009), analyzing effects of attention and awareness (Wyart, Tallon-Baudry, 2009). She has directly tested, with behavior and MEG recording, implications of hierarchy and reverse hierarchy theories, including global information processing being first and mandatory in conscious perception (Campana, Tallon-Baudry, 2013; Campana … Tallon-Baudry, 2016) In summary, bottom-up versus top-down processing theories reflect on the essence of perception: the dichotomy of rapid vision-at-a-glance versus slower vision-with-scrutiny, roles of attention, hierarchy of visual representation levels, roles of feedback connections, sites and mechanisms of various visual phenomena, and sources of perceptual/cognitive deficits (Neglect, Dyslexia, ASD). Speakers at the proposed symposium will address these issues with both a historical and forward looking perspective.

Presentations

Is Guided Search 6.0 compatible with Reverse Hierarchy Theory

Jeremy M Wolfe¹; ¹Harvard Medical School and Visual Attention Lab Brigham & Women’s Hospital

It has been 30 years since the first version of the Guided Search (GS) model of visual search was published. As new data about search accumulated, GS needed modification. The latest version is GS6. GS argues that visual processing is capacity-limited and that attention is needed to “bind” features together into recognizable objects. The core idea of GS is that the deployment of attention is not random but is “guided” from object to object. For example, in a search for your black shoe, search would be guided toward black items. Earlier versions of GS focused on top-down (user-driven) and bottom-up (salience) guidance by basic features like color. Subsequent research adds guidance by history of search (e.g. priming), value of the target, and, most importantly, scene structure and meaning. Your search for the shoe will be guided by your understanding of the scene, including some sophisticated information about scene structure and meaning that is available “preattentively”. In acknowledging the initial, preattentive availability of something more than simple features, GS6 moves closer to ideas that are central to the Reverse Hierarchy Theory of Hochstein and Ahissar. As is so often true in our field, this is another instance where the answer is not Theory A or Theory B, even when they seem diametrically opposed. The next theory tends to borrow and synthesize good ideas from both predecessors.

Gist perception precedes awareness of details in various tasks and populations

Shaul Hochstein¹; ¹Life Sciences, Hebrew University, Jerusalem

Reverse Hierarchy Theory proposes several dramatic propositions regarding conscious visual perception. These include the suggestion that, while the visual system receives scene details and builds from them representations of the objects, layout, and structure of the scene, nevertheless, the first conscious percept is that of the gist of the scene – the result of implicit bottom-up processing. Only later does conscious perception attain scene details by return to lower cortical area representations. Recent studies at our lab analyzed phenomena whereby participants receive and perceive the gist of the scene before and without need for consciously knowing the details from which the gist is constructed. One striking conclusion is that “pop-out” is an early high-level effect, and is therefore not restricted to basic element features. Thus, faces pop-out from heterogeneous objects, and participants are unaware of rejected objects. Our recent studies of ensemble statistics perception find that computing set mean does not require knowledge of its individuals. This mathematically-improbable computation is both useful and natural for neural networks. I shall discuss just how and why set means are computed without need for explicit representation of individuals. Interestingly, our studies of neglect patients find that their deficit is in terms of tasks requiring focused attention to local details, and not for those requiring only global perception. Neglect patients are quite good at pop-out detection and include left-side elements in ensemble perception.

From global to local in conscious vison: behavior & MEG

Catherine Tallon-Baudry¹; ¹CNRS Cognitive Neuroscience, Ecole Normale Supérieure, Paris

The reverse hierarchy theory makes strong predictions on conscious vision. Local details would be processed in early visual areas before being rapidly and automatically combined into global information in higher order area, where conscious percepts would initially emerge. The theory thus predicts that consciousness arises initially in higher order visual areas, independently from attention and task, and that additional and optional attentional processes operating from top to bottom are needed to retrieve local details. We designed novel textured stimuli that, as opposed to Navon’s letters, are truly hierarchical. Taking advantage of both behavioral measures and of the decoding of MEG data, we show that global information is consciously perceived faster than local details, and that global information is computed regardless of task demands during early visual processing. These results support the idea that global dominance in conscious percepts originates in the hierarchical organization of the visual system. Implications for the nature of conscious visual experience and its underlying neural mechanisms will be discussed.

Next-generation models of recurrent computations in the ventral visual stream

James DiCarlo¹; ¹Neuroscience, McGovern Inst. & Brain & Cognitive Sci., MIT

Understanding mechanisms underlying visual intelligence requires combined efforts of brain and cognitive scientists, and forward engineering emulating intelligent behavior (“AI engineering”). This “reverse-engineering” approach has produced more accurate models of vision. Specifically, a family of deep artificial neural-network (ANN) architectures arose from biology’s neural network for object vision — the ventral visual stream. Engineering advances applied to this ANN family produced specific ANNs whose internal in silico “neurons” are surprisingly accurate models of individual ventral stream neurons, that now underlie artificial vision technologies. We and others have recently demonstrated a new use for these models in brain science — their ability to design patterns of light energy images on the retina that control neuronal activity deep in the brain. The reverse engineering iteration loop — respectable ANN models to new ventral stream data to even better ANN models — is accelerating. My talk will discuss this loop: experimental benchmarks for in silico ventral streams, key deviations from the biological ventral stream revealed by those benchmarks, and newer in silico ventral streams that partly close those differences. Recent experimental benchmarks argue that automatically-evoked recurrent processing is critically important to even the first 300msec of visual processing, implying that conceptually simpler, feedforward only, ANN models are no longer tenable as accurate in silico ventral streams. Our broader aim is to nurture and incentivize next generation models of the ventral stream via a community software platform termed “Brain-Score” with the goal of producing progress that individual research groups may be unable to achieve.

Visual and non-visual skill acquisition – success and failure

Merav Ahissar¹; ¹Psychology Department, Social Sciences & ELSC, Hebrew University, Israel

Acquiring expert skills requires years of experience – whether these skills are visual (e.g. face identification), motor (playing tennis) or cognitive (mastering chess). In 1977, Shiffrin & Schneider proposed an influential stimulus-driven, bottom-up theory of expertise automaticity, involving mapping stimuli to their consistent response. Integrating many studies since, I propose a general, top-down theory of skill acquisition. Novice performance is based on the high-level multiple-demand (Duncan, 2010) fronto-parietal system, and with practice, specific experiences are gradually represented in lower-level domain-specific temporal regions. This gradual process of learning-induced reverse-hierarchies is enabled by detection and integration of task-relevant regularities. Top-down driven learning allows formation of task-relevant mapping and representations. These in turn form a space which affords task-consistent interpolations (e.g. letters in a manner crucial for letter identification rather than visual similarity). These dynamics characterize successful skills. Some populations, however, have reduced sensitivity to task-related regularities, hindering their related skill acquisition, preventing specific expertise acquisition even after massive training. I propose that skill-acquisition failure, perceptual as cognitive, reflects specific difficulties in detecting and integrating task-relevant regularities, impeding formation of temporal-area expertise. Such is the case for individuals with dyslexia (reduced retention of temporal regularities; Jaff-Dax et al., 2017), who fail to form an expert visual word-form area, and for individuals with autism (who integrate regularities too slowly for online updating; Lieder et al., 2019). Based on this general conceptualization, I further propose that this systematic impediment.

< Back to 2021 Symposia