2018 Davida Teller Award – Nancy Kanwisher

Vision Sciences Society is honored to present Dr. Nancy Kanwisher with the 2018 Davida Teller Award

VSS established the Davida Teller Award in 2013. Davida was an exceptional scientist, mentor and colleague, who for many years led the field of visual development. The award is therefore given to an outstanding woman vision scientist with a strong history of mentoring.

Nancy Kanwisher

Walter A. Rosenblith Professor, Department of Brain and Cognitive Sciences, McGovern Institute for Brain Research, Massachusetts Institute of Technology

Functional imaging of the brain as a window into the architecture of the human mind

Dr. Kanwisher will talk during the Awards Session
Monday, May 21, 2018, 12:30 – 1:30 pm, Talk Room 1-2

The last twenty years of fMRI research have given us a new sketch of the human mind, in the form of the dozens of cortical regions that have now been identified, many with remarkably specific functions. I will describe several ongoing lines of work in my lab on cortical regions engaged in perceiving social interactions, understanding the physical world, and perceiving music. After presenting various findings that use pattern analysis (MVPA), I will also raise caveats about this method, which can both fail to reveal information that we know is present in a given region, and which can also reveal information that is likely epiphenomenal. I’ll argue that  human cognitive neuroscience would greatly benefit from the invention of new tools to address these challenges.


My research uses fMRI and other methods to try to discover the functional organization of the brain as a window into the architecture of the human mind. My early forays in this work focused on high-level visual cortex, where my students and I developed the methods to test the functional profile of regions in the ventral visual pathway specialized for the perception of face, places, bodies, and words. The selectivity of these regions is now widely replicated, and ongoing work  in my lab and many other labs is now asking what exactly is represented and computed in each of these regions, how they arise both developmentally and evolutionarily, how they are structurally connected to each other and the rest of the brain, what the causal role of each is in behavior and perceptual awareness, and why, from a computational point of view, we have functional selectivity in the brain in the first place.

My career would quite simply never have happened without the great gift of fabulous mentors. Molly Potter fought to have me accepted to graduate school (from the bottom of the waiting list), and, against all reason, did not give up on me even when I dropped out of grad school three times to try to become a journalist.  Then after a diversionary postdoc in international security,  Anne Treisman gave me an incredible second chance in vision research as a postdoc in her lab, despite my scanty list of publications. Later in my own lab, my luck came in the form of spectacular mentees. I have had the enormous privilege and delight of working with many of the most brilliant young scientists in my field.

I think we scientists have an obligation to share the cool results of our work with the public (who pays for it). My latest effort in this direction is my growing collection of short lectures about human cognitive neuroscience for lay and undergraduate audiences: nancysbraintalks.mit.edu.

2018 Satellite Events

Wednesday, May 16

Computational and Mathematical Models in Vision (MODVIS)

Wednesday, May 16 – Friday, May 18, Horizons
9:00 am – 6:00 pm, Wednesday
9:00 am – 6:00 pm, Thursday
8:30 – 11:45 am Friday

Organizers: Jeff Mulligan, NASA Ames Research Center; Zygmunt Pizlo, UC Irvine; Anne B. Sereno, Purdue University; and Qasim Zaidi, SUNY College of Optometry

Keynote Selection Committee: Yalda Mohsenzadeh, MIT; Michael Rudd, University of Washington

The 7th VSS satellite workshop on Computational and Mathematical Models in Vision (MODVIS) will be held at the Tradewinds Island Resorts in St. Pete Beach, FL, May 16 – May 18. A keynote address will be given by Eero Simoncelli, New York University.

The early registration fee is $100 for regular participants, $50 for students. More information can be found on the workshop’s website: http://www.conf.purdue.edu/modvis/

Thursday, May 17

Eye Tracking in Virtual Reality

Thursday, May 17, 10:00 am – 3:00 pm, Jasmine/Palm

Organizer: Gabriel Diaz, Rochester Institute of Technology

This will be a hands-on workshop run by Gabriel Diaz, with support from his graduate students Kamran Binaee and Rakshit Kothari.

The ability to incorporate eye tracking into computationally generated contexts presents new opportunities for research into gaze behavior. The aim of this workshop is to provide an understanding of the hardware, data collection process, and algorithms for data analysis. Example data and code will be provided in two both Jupyter notebooks and Matlab (choose your preference). This workshop is sponsored by The Optical Society’s Vision Technical Group and is suitable for both PIs and graduate students.

Friday, May 18

Tutorial on Big Data and Online Crowd-Sourcing for Vision Research

Friday, May 18, 8:30 – 11:45 am, Jasmine/Palm

Organizer: Wilma Bainbridge, National Institutes of Health

Speakers: Wilma Bainbridge, National Institutes of Health; Tim Brady, University of California San Diego; Dwight Kravitz, George Washington University; and Gijsbert Stoet, Leeds Beckett University

Online experiments and Big Data are becoming big topics in the field of vision science, but can be hard to access for people not familiar with web development and coding. This tutorial will teach attendees the basics of creating online crowd-sourced experiments, and how to think about collecting and analyzing Big Data related to vision research. Four experts in the field will discuss how they use and collect Big Data, and give hands-on practice to tutorial attendees. We will discuss Amazon Mechanical Turk, its strengths and weaknesses, and how to leverage it in creative ways to collect powerful, large-scale data. We will then discuss Psytoolkit, an online experimental platform for coding timed behavioral and psychophysical tasks, that can integrate with Amazon Mechanical Turk. We will then discuss how to create Big Datasets using various ways of “scraping” large-scale data from the internet. Finally, we will discuss other sources of useful crowd-sourced data, such as performance on mobile games, and methods for scaling down and analyzing these large data sets.

To help us plan for this event, please register here: http://wilmabainbridge.com/research/bigdata/bigdataregistration.html

Sunday, May 20

FoVea (Females of Vision et al) Workshop

Sunday, May 20, 7:30 – 8:30 pm, Horizons

Organizers: Diane Beck, University of Illinois, Urbana-Champaign; Mary A. Peterson, University of Arizona; Karen Schloss, University of Wisconsin – Madison; Allison Sekuler, Baycrest Health Sciences

Speaker: Virginia Valian, Hunter College
Title: Remedying the (Still) Too Slow Advancement of Women

Dr. Valian is a Distinguished Professor of Psychology and Director of The Gender Equity Project.

FoVea is a group founded to advance the visibility, impact, and success of women in vision science (www.foveavision.org). We encourage vision scientists of all genders to participate in the workshops.

Please register at: http://www.foveavision.org/vss-workshops

Monday, May 21

Psychophysics Toolbox Discussion

Monday, May 21, 2:00 – 3:00 pm, Talk Room 1

Organizer: Vijay Iyer, MathWorks

Panelists: Vijay Iyer, David Brainard, and Denis Pelli

Discussion of the current-state (technical, funding, community status) of the Psychophysics toolbox, widely used for visual stimulus generation in vision science experiments.

Social Hour for Faculty at Primarily Undergraduate Institutions (PUIs)

Monday, May 21, 2:00 – 4:00 pm, Royal Tern

Organizer: Katherine Moore, Arcadia University

Do you work at a primarily undergraduate institution (PUI)? Do you juggle your research program, student mentoring, and a heavy teaching load? If so, come along to the PUI social and get to know other faculty at PUIs! It will be a great opportunity to share your ideas and concerns. Feel free to bring your own drinks / snacks. Prospective faculty of PUIs are also welcome to attend and get to know us and our institutions.

Canadian Vision Social

Monday, May 21, 2:00 – 4:00 pm, Jasmine/Palm

Organizer: Doug Crawford, York Centre for Vision Research

This afternoon Social is open to any VSS member who is, knows, or would like to meet a Canadian Vision Scientist! This event will feature free snacks and refreshments, with a complementary beverage for the first 200 attendees. We particularly encourage trainees and scientists who would like to learn about the various research and training funds available through York’s Vision: Science to Applications (VISTA) program. This event is sponsored by the York Centre for Vision Research and VISTA, which is funded in part by the Canada First Research Excellence Fund (CFREF).

Tuesday, May 22

Virtual Reality as a Tool for Vision Scientists

Tuesday, May 22, 1:00 – 2:00 pm, Talk Room 1
Organizer: Matthias Pusch, WorldViz

In a hands on group session, we will show how Virtual Reality can be used by Vision Scientists for remote and on site collaborative experiments. Full experimental control over stimuli and reactions enable a unique setting for measuring performance. We will experience collaboration with off-site participants, and show the basics of performance data recording and analysis.

2018 Student Workshops

There is no advanced sign-up for workshops. Workshops will be filled on a first-come, first-served basis.

VSS Workshop for PhD Students and Postdocs:
Getting that Faculty Job

Saturday, May 19, 2018, 1:00 – 2:00 pm, Jasmine/Palm
Moderator: David Brainard
Panelists: Michelle Greene, Tim Brady, Nicole Rust, James Elder

A key transition on the academic career path is obtaining a faculty position.  This workshop will focus on the application process (optimizing CV, statements, letters), the interview and job talk, handling the two-body problem, and post-offer steps such as negotiation about start-up funds, space, and teaching responsibilities.  Panelists include junior scientists who have recently obtained a faculty position as well as more senior scientists who can offer perspective from the hiring side of the process.

Michelle Greene, Bates College
Michelle R. Greene is an Assistant Professor of Neuroscience at Bates College, where she heads the Bates Computational Vision Laboratory. Her work examines the temporal evolution of high-level visual perception. She received her PhD from MIT in 2009, and did postdoctoral work at Harvard Medical School and Stanford University before joining Bates in 2017.
Tim Brady, UCSD
Timothy Brady is an Asst. Professor in the Department of Psychology at the University of California, San Diego, where he started in 2015, ending his need to think about the faculty job market forever (he hopes). His research uses a combination of behavioral, computational and cognitive neuroscience methods to understand the limits on our ability to encode and maintain information in visual memory. He received his B.A. in Cognitive Science from Yale University ’06, his Ph.D. from MIT in Brain and Cognitive Sciences ’11 and conducted postdoctoral research in the Harvard University Vision Sciences Laboratory ’11-’15.
Nicole Rust, University of Pennsylvania
Nicole Rust is an Associate Professor in the Department of Psychology.  She received her Ph.D. in neuroscience from New York University, and trained as a postdoctoral researcher at Massachusetts Institute of Technology before joining the faculty at Penn in 2009. Research in her laboratory is focused on understanding the neural basis of visual memory, including our remarkable ability to remember the objects and scenes that we have encountered, even after viewing thousands, each only for few seconds. To understand visual memory, her lab employs a number of different approaches, including investigations of human and animal visual memory behaviors, measurements and manipulations of neural activity, and computational modeling. She has received a number of awards for both research and teaching including a McKnight Scholar award, an NSF CAREER award, a Alfred P. Sloan Fellowship, and the Charles Ludwig Distinguished teaching award. Her research is currently funded by the National Eye Institute at the National Institutes of Health, the National Science Foundation, and the Simons Collaboration on the Global Brain.
James Elder, York University
James Elder is a Professor in the Department of Psychology and the Department of Electrical Engineering & Computer Science at York University, and a member of York’s Centre for Vision Research and Vision:  Science to Applications (VISTA) program. His research seeks to improve machine vision systems through a better understanding of visual processing in biological systems. Dr. Elder’s current research is focused on natural scene statistics, perceptual organization, contour processing, shape perception, single-view 3D reconstruction, attentive vision systems and machine vision systems for dynamic 3D urban awareness.
David Brainard, University of Pennsylvania
David H. Brainard is the RRL Professor of Psychology at the University of Pennsylvania. He is a fellow of the Optical Society, ARVO and the Association for Psychological Science. At present, he directs Penn’s Vision Research Center, co-directs Penn’s Computational Neuroscience Initiative, co-directs Penn’s NSF funded certificate program in Complex Scene Perception, is on the Board of the Vision Sciences Society, and is a member of the editorial board of the Journal of Vision. His research interests focus on human color vision, which he studies both experimentally and through computational modeling of visual processing. He will be moderating this session.

VSS Workshop for PhD Students and Postdocs:
The public face of your science

Sunday, May 20, 2018, 1:00 – 2:00 pm, Jasmine/Palm
Moderator: Jeff Schall
Panelists: Allison Sekuler, Frans Verstraten, Morgan Ryan

Your research has several potential audiences. In this workshop, we will focus on the general public. When should you tell the world about your latest results? Always? Only if you think it is particularly noteworthy? Only when someone else asks? How should you communicate with the public? Social media? Press releases? How can you attract attention for your work (when you want to) and what should you do if you attract attention that you do not want? Our panel consists of two vision scientists, Allison Sekuler and Frans Verstraten, who have experience in the public eye and Morgan Ryan, the editor for SpringerNature, who handles the Psychonomic Society journals (including AP&P, PBR, and CRPI). Bring your questions.

Allison Sekuler, McMaster University
Dr. Allison Sekuler is Vice-President of Research and the Sandra A. Rotman Chair at Baycrest Health Sciences. She came to Baycrest from her position as a Professor in the Department of Psychology, Neuroscience & Behaviour at McMaster University, where she was the first Canada Research Chair in Cognitive Neuroscience (2001-2011). She is also the Co-Chair of the Academic Colleagues at the Council of Ontario Universities and Chair of the Natural Sciences and Engineering Research Council of Canada‘s (NSERC) Scholarships & Fellowships group along with being a member of NSERC’s Committee for Discovery Research. The recipient of numerous awards for research, teaching and leadership, Dr. Sekuler has a notable record of scientific achievements in aging and vision science, cognitive neuroscience, learning and neural plasticity, and neuroimaging and neurotechnology, as well as extensive experience in senior academic and research leadership roles.
Frans Verstraten, University of Sydney
Professor Frans Verstraten is the McCaughey Chair of Psychology at the University of Sydney and Head of School. He was a board member and former president of the Vision Sciences Society. Before his move to Australia in 2012 he was also active in the domains of the popularization of science and science communication. Among other things, he gave many talks for the general audience, participated in a popular science TV-show for several years, and wrote columns in a national newspaper and several magazines. He has been a member of many national and international committees where he represents the psychological and behavioural sciences. Currently, he tries to convince the University’s marketing and communication teams to understand the power of good press releases (and to refrain from making unwarranted statements to spice research results up).
Morgan Ryan, SpringerNature
With over eight years of experience in scholarly publishing, Morgan Ryan is a Senior Editor in Behavioral Sciences at Springer, part of Springer Nature. As the Publishing Development Editor for more than 14 psychology journals, including the Psychonomic Society journals, she has extensive experience in research promotion and journal strategy. Among other projects, she has organized and presented research-publishing workshops for graduate students and early career scholars.  She enjoys initiating and coordinating press office activity between Springer and the Psychonomic Society  to increase the public visibility of science.
Jeff Schall, Vanderbilt University
The session will be moderated by Jeff Schall, who is the E. Bronson Ingram Professor of Neuroscience and Professor of Psychology and of Ophthalmology & Visual Sciences at Vanderbilt University. Schall’s research investigates how the visual system selects targets for and controls the initiation of saccades using cognitive neurophysiology, anatomical and computational approaches. Schall is a founding member of the advisory board for the interdisciplinary major at Vanderbilt, Communication of Science and Technology, through which students master communication tools and techniques, learn science, and are embedded in research programs. He has also been involved in the complexities of communication at the boundary of law and neuroscience.



Clinical insights into basic visual processes

Time/Room: Friday, May 18, 2018, 12:00 – 2:00 pm, Talk Room 1
Organizer(s): Paul Gamlin, University of Alabama at Birmingham; Ann E. Elsner, Indiana University; Ronald Gregg, University of Louisville
Presenters: Geunyoung Yoon, Artur Cideciyan, Ione Fine, MiYoung Kwon

< Back to 2018 Symposia

Symposium Description

This year’s biennial ARVO at VSS symposium features insights into human visual processing at the retinal and cortical level arising from clinical and translational research. The speakers will present recent work based on a wide range of state-of-the art techniques including adaptive optics, brain and retinal imaging, psychophysics and gene therapy.


Neural mechanisms of long-term adaptation to the eye’s habitual aberration

Speaker: Geunyoung Yoon, Flaum Eye Institute, Center for Visual Science, The Institute of Optics, University of Rochester

Understanding the limits of human vision requires fundamental insights into both optical and neural factors in vision. Although the eye’s optics are far from perfect, contributions of the optical factors to neural processing are largely underappreciated. Specifically, how neural processing of images formed on the retina is altered by the long-term visual experience with habitual optical blur has remained unexplored. With technological advances in an adaptive optics vision simulator, it is now possible to manipulate ocular optics precisely. I will highlight our recent investigations on underlying mechanisms of long-term neural adaptation to the optics of the eye and its impact on spatial vision in the normally developed adult visual system.

Human Melanopic Circuit in Isolation from Photoreceptor Input: Light Sensitivity and Temporal Profile

Speaker: Artur Cideciyan, Scheie Eye Institute, Perelman School of Medicine, University of Pennsylvania

Leber congenital amaurosis refers to a group of severe early-onset inherited retinopathies. There are more than 20 causative genes with varied pathophysiological mechanisms resulting in vision loss at the level of the photoreceptors. Some eyes retain near normal photoreceptor and inner retinal structure despite the severe retina-wide loss of photoreceptor function. High luminance stimuli allow recording of pupillary responses driven directly by melanopsin-expressing intrinsically photosensitive retinal ganglion cells. Analyses of these pupillary responses help clarify the fidelity of transmission of light signals from the retina to the brain for patients with no light perception undergoing early phase clinical treatment trials. In addition, these responses serve to define the sensitivity and temporal profile of the human melanopic circuit in isolation from photoreceptor input.

Vision in the blind

Speaker: Ione Fine, Department of Psychology, University of Washington

Individuals who are blind early in life show cross-modal plasticity – responses to auditory and tactile stimuli within regions of occipital cortex that are purely visual in the normally sighted. If vision is restored later in life, as occurs in a small number of sight recovery individuals, this cross-modal plasticity persists, even while some visual responsiveness is regained. Here I describe the relationship between cross-modal responses and persisting residual vision. Our results suggest the intriguing possibility that the dramatic changes in function that are observed as a result of early blindness are implemented in the absence of major changes in neuroanatomy at either the micro or macro scale: analogous to reformatting a Windows computer to Linux.

Impact of retinal ganglion cell loss on human pattern recognition

Speaker: MiYoung Kwon, Department of Ophthalmology, University of Alabama at Birmingham

The processing of human pattern detection and recognition requires integrating visual information across space. In the human visual system, the retinal ganglion cells (RGCs) are the output neurons of the retina, and human pattern recognition is built from the neural representation of the RGCs. Here I will present our recent work demonstrating how a loss of RGCs due to either normal aging or pathological conditions such as glaucoma undermines pattern recognition and alters spatial integration properties. I will further highlight the role of the RGCs in determining the spatial extent over which visual inputs are combined. Our findings suggest that understanding the structural and functional integrity of RGCs would help not only better characterize visual deficits associated eye disorders, but also understand the front-end sensory requirements for human pattern recognition.

< Back to 2018 Symposia

Visual remapping: From behavior to neurons through computation

Time/Room: Friday, May 18, 2018, 5:00 – 7:00 pm, Talk Room 1
Organizer(s): James Mazer, Cell Biology & Neuroscience, Montana State University, Bozeman, MT & Fred Hamker, Chemnitz University of Technology, Chemnitz, Germany
Presenters: Julie Golomb, Patrick Cavanagh, James Bisley, James Mazer, Fred Hamker

< Back to 2018 Symposia

Symposium Description

Active vision in both humans and non-human primates depends on saccadic eye movements to accurately direct the foveal portion of the retina towards salient visual scene features. Saccades, in concert with visual attention, can faciliate efficient allocation of limited neural and computational resources in the brain during visually guided behaviors. Saccades, however, are not without consequences; saccades can dramatically alter the spatial distribution of activity in the retina several times per second. This can lead to large changes to the cortical scene representation even when the scene is static. Behaviors that depend on accurate visuomotor coordination and stable sensory (and attentional) representations in the brain, like reaching and grasping, must somehow compensate for the apparent scene changes caused by eye movements. Recent psychophysical, neurophysiological and modeling results have shed new light on the neural substrates of this compensatory process. Visual “remapping” has been identified as a putative mechanism for stabilizing visual and attentional representations across saccades. At the neuronal level, remapping occurs when neuronal receptive fields shift in anticipation of a saccade, as originally described in the lateral intraparietal area of the monkey (Duhamel et al., 1992). It has been suggested that remapping facilitates perceptual stability by bridging pre- and post-saccadic visual and attentional representations in the brain. In this symposium we will address the functional role of remapping and the specific relationship between neurophysiological remapping (a single-neuron phenomenon) and psychophysically characterized perisaccadic changes in visual perception and attentional facilitation. We propose to consider computational modeling as a potential bridge to connect these complementary lines of research. The goal of this symposium is to clarify our current understanding of physiological remapping as it occurs in different interconnected brain regions in the monkey (V4, LIP and FEF) and to address how remapping at the neuronal level can account for observed perisaccadic changes in visual perception and attentional state. Symposium participants have been drawn from three different, yet complementary, disciplines: psychophysics, neurophysiology and computational modeling. Their approaches have provided novel insights into remapping at phenomenological, functional and mechanistic levels. Remapping is currently a major area of research in all three disciplines and, while there are several common themes developing, there remains substantial debate about the degree to which remapping can account for various psychophysical phenonomena. We propose that bringing together key researchers using different approaches to discuss the implications of currently available data and models will both advance our understanding of remapping and be broad interest to VSS members (both students and faculty) across disciplines.


Remapping of object features: Implications of the two-stage theory of spatial remapping

Speaker: Julie Golomb, The Ohio State University, Columbus, OH

When we need to maintain spatial information across an eye movement, it is an object’s location in the world, not its location on our retinas, which is generally relevant for behavior. A number of studies have demonstrated that neurons can rapidly remap visual information, sometimes even in anticipation of an eye movement, to preserve spatial stability. However, it has also been demonstrated that for a period of time after each eye movement, a “retinotopic attentional trace” still lingers at the previous retinotopic location, suggesting that remapping actually manifests in two overlapping stages, and may not be as fast or efficient as previously thought. If spatial attention is remapped imperfectly, what does this mean for feature and object perception? We have recently demonstrated that around the time of an eye movement, feature perception is distorted in striking ways, such that features from two different locations may be simultaneously bound to the same object, resulting in feature-mixing errors. We have also revealed that another behavioral signature of object-location binding, the “spatial congruency bias”, is tied to retinotopic coordinates after a saccade. These results suggest that object-location binding may need to be re-established following each eye movement rather than being automatically remapped. Recent efforts from the lab are focused on linking these perceptual signatures of remapping with model-based neuroimaging, using fMRI multivoxel pattern analyses, inverted encoding models, and EEG steady-state visual evoked potentials to dynamically track both spatial and feature remapping across saccades.

Predicting the present: saccade based vs motion-based remapping

Speaker: Patrick Cavanagh, Glendon College, Toronto, ON and Dartmouth College, Hanover, NH

Predictive remapping alerts neurons when a target will fall into its receptive field after an upcoming saccade. This has consequences for attention which starts selecting information from the target’s remapped location before the eye movement begins even though that location is not relevant to pre-saccadic processing. Thresholds are lower and information from the target’s remapped and current locations may be integrated. These predictive effects for eye movements are mirrored by predictive effects for object motion, in the absence of saccades: motion-based remapping. An object’s motion is used to predict its current location and as a result, we sometimes see a target far from its actual location: we see it where it should be now. However, these predictions operate differently for eye movements and for perception, establishing two distinct representations of spatial coordinates. We have begun identifying the cortical areas that carry these predictive position representations and how they may interface with memory and navigation.

How predictive remapping in LIP (but not FEF) might explain the illusion of perceptual stability

Speaker: James Bisley, Department of Neurobiology, David Geffen School of Medicine at UCLA, Los Angeles, California

The neurophysiology of remapping has tended to examine the latency of responses to stimuli presented around a single saccade. Using a visual foraging task, in which animals make multiple eye movements within a trial, we have examined predictive remapping in the lateral intraparietal area (LIP) and the frontal eye field (FEF) with a focus on when activity differentiates between stimuli that are brought on to the response field. We have found that the activity in LIP, but not FEF, rapidly shifts from a pre-saccadic representation to a post-saccadic representation during the period of saccadic suppression. We hypothesize that this sudden switch keeps attentional priorities of high priority locations stable across saccades and, thus, could create the illusion of perceptual stability.

Predictive attentional remapping in area V4 neurons

Speaker: James Mazer, Cell Biology & Neuroscience, Montana State University, Bozeman, MT

Although saccades change the distribution of neural activity throughout the visual system, visual perception and spatial attention are relatively unaffected by saccades. Studies of human observers have suggested that attentional topography in the brain is stablized across saccades by an active process that redirects attentional facilitation to the right neurons in retinotopic visual cortex. To characterize the specific neuronal mechanisms underlying this retargeting process we trained two monkeys to perform a novel behavioral task that required them to sustain attention while making guided saccades. Behavioral performance data indicate that monkeys, like humans, can sustain spatiotopic attention across saccades. Data recorded from neurons in extrastriate area V4 during task performance were used to access perisaccadic attentional dynamics. Specificially, we asked when attentional facilitation turns on or off relative to saccades and how attentional modulation changes depending on whether a saccade brings a neuron’s receptive field (RF) into or out of the attended region. Our results indicate that for a substantial fraction of V4 neurons, attentional state changes begin ~100 ms before saccade onset, consistent with the timing of predictive attentional shifts in human observers measured psychophysically. In addition, although we found little evidence of classical, LIP-style spatial remapping in V4, there was a small anticipatory shift or skew of the RF in the 100ms immediately saccades detectable at the population level, although it is unclear of this effect corresponds to a shift towards the saccade endpoint or reflects a shift parallel to the saccade vector.

Neuro-computational models of spatial updating

Speaker: Fred Hamker, Chemnitz University of Technology, Chemnitz, Germany

I review neuro-computational models of peri-saccadic spatial perception that provide insight into the neural mechanisms of spatial updating around eye movements. Most of the experimental observations can be explained by only two different models, one involves spatial attention directed towards the saccade target and the other relies on predictive remapping and gain-fields for coordinate transformation. The latter model uses two eye related signals: a predictive corollary discharge and eye position, which updates after saccade. While spatial attention is mainly responsible for peri-saccadic compression, predictive remapping (in LIP) and gain-fields for coordinate transformation can account for the shift of briefly flashed bars in total darkness and for the increase of the threshold in peri-saccadic displacement detection. With respect to the updating of sustained spatial attention, recently, two different types were discovered. One study shows that attention lingers after saccade at the (irrelevant) retinotopic position, another shows that shortly before saccade onset, spatial attention is remapped to a position opposite to the saccade direction. I show new results which demonstrate that both observations are not contradictory and emerge through model dynamics: The lingering of attention is explained by the (late-updating) eye position signal, which establishes an attention pointer in an eye-reference frame. This reference shifts with the saccade and updates attention to the initial position only after saccade. The remapping of attention opposite to the saccade direction is explained by the corollary discharge signal, which establishes a transient eye-reference frame, anticipates the saccade and thus updates attention prior to saccade onset.

< Back to 2018 Symposia

Prediction in perception and action

Time/Room: Friday, May 18, 2018, 2:30 – 4:30 pm, Talk Room 1
Organizer(s): Katja Fiehler, Department of Psychology and Sports Science, Giessen University, Giessen, Germany
Presenters: Mary Hayhoe, Miriam Spering, Cristina de la Malla, Katja Fiehler, Kathleen Cullen

< Back to 2018 Symposia

Symposium Description

Prediction is an essential mechanism enabling humans to prepare for future events. This is especially important in a dynamically changing world, which requires rapid and accurate responses to external stimuli. Predictive mechanisms work on different time scales and at various information processing stages. They allow us to anticipate the future state both of the environment and ourselves. They are instrumental to compensate for noise and delays in the transmission of neural signals and allow us to distinguish external events from the sensory consequences of our own actions. While it is unquestionable that predictions play a fundamental role in perception and action, their underlying mechanisms and neural basis are still poorly understood. The goal of this symposium is to integrate recent findings from psychophysics, sensorimotor control, and electrophysiology to update our current understanding of predictive mechanisms in different sensory and motor systems. It brings together a group of leading scientists at different stages in their career who all have made important contributions to this topic. Two prime examples of predictive processes are considered: when interacting with moving stimuli and during self-generated movements. The first two talks from Hayhoe and Spering will focus on the oculomotor system which provides an excellent model for examining predictive behavior. They will show that smooth pursuit and saccadic eye movements significantly contribute to sucessful predictions of future visual events. Moreover, Hayhoe will provide examples for recent advances in the use of virtual reality (VR) techniques to study predictive eye movements in more naturalistic situations with unrestrained head and body movements. De la Malla will extend these findings to the hand movement system by examining interceptive manual movements. She will conclude that predictions are continuously updated and combined with online visual information to optimize behavior. The last two talks from Fiehler and Cullen will take a different perspective by considering predictions during self-generated movements. Such predictive mechanims have been associated with a forward model that predicts the sensory consequences of our own actions and cancels the respective sensory reafferences. Fiehler will focus on such cancellation mechanisms and present recent findings on tactile suppression during hand movements. Based on electrophysiological studies on self-motion in monkeys, Cullen will finally answer where and how the brain compares expected and actual sensory feedback. In sum, this symposium targets the general VSS audience and aims to provide a novel and comprehensive view on predictive mechanisms in perception and action spanning from behavior to neurons and from strictly laboratory tasks to (virtual) real world scenarios.


Predictive eye movements in natural vision

Speaker: Mary Hayhoe, Center for Perceptual Systems, University of Texas Austin, USA

Natural behavior can be described as a sequence of sensory motor decisions that serve behavioral goals. To make action decisions the visual system must estimate current world state. However, sensory-motor delays present a problem to a reactive organism in a dynamically changing environment. Consequently it is advantageous to predict future state as well. This requires some kind of experience-based model of how the current state is likely to change over time. It is commonly accepted that the proprioceptive consequences of a planned movement are predicted ahead of time using stored internal models of the body’s dynamics. It is also commonly assumed that prediction is a fundamental aspect of visual perception, but the existence of visual prediction and the particular mechanisms underlying such prediction are unclear. Some of the best evidence for prediction in vision comes from the oculomotor system. In this case, both smooth pursuit and saccadic eye movements reveal prediction of the future visual stimulus. I will review evidence for prediction in interception actions in both real and virtual environments. Subjects make accurate predictions of visual target motion, even when targets follow trajectories determined by the complex dynamics of physical interactions, and the head and body are unrestrained. These predictions appear to be used in common by both eye and arm movements. Predictive eye movements reveal that the observer’s best guess at the future state of the environment is based on image data in combination with representations that reflect learnt statistical properties of dynamic visual environments.

Smooth pursuit eye movements as a model of visual prediction

Speaker: Miriam Spering, Department of Ophthalmology & Visual Sciences, University of British Columbia, Vancouver, Canada

Real-world movements, ranging from intercepting prey to hitting a ball, require rapid prediction of an object’s trajectory from a brief glance at its motion. The decision whether, when and where to intercept is based on the integration of current visual evidence, such as the perception of a ball’s direction, spin and speed. However, perception and decision-making are also strongly influenced by past sensory experience. We use smooth pursuit eye movements as a model system to investigate how the brain integrates sensory evidence with past experience. This type of eye movement provides a continuous read-out of information processing while humans look at a moving object and make decisions about whether and how to interact with it. I will present results from two different series of studies: the first utilizes anticipatory pursuit as a means to understand the temporal dynamics of prediction, and probes the modulatory role of expectations based on past experience. The other reveals the benefit of smooth pursuit itself, in tasks that require the prediction of object trajectories for perceptual estimation and manual interception. I will conclude that pursuit is both an excellent model system for prediction, and an important contributor to successful prediction of object motion.

Prediction in interceptive hand movements

Speaker: Cristina de la Malla, Department of Human Movement Sciences, Vrije Universiteit Amsterdam, The Netherlands

Intercepting a moving target requires spatial and temporal precision: the target and the hand need to be at the same position at the same time. Since both the target and the hand move, we cannot just aim for the target’s current position, but need to predict where the target will be by the time we reach it. We normally continuously track targets with our gaze, unless the characteristics of the task or of the target make it impossible to do so. Then, we make saccades and direct our movements towards specific locations where we predict the target will be in the future. If the precise location at which one is to hit the target only becomes evident as the target approaches the interception area, the gaze, head and hand movements towards this area are delayed due to not having the possibility of predicting the target future position. Predictions are continuously updated and combined with online visual information to optimize our actions: the less predictable the target’s motion, the more we have to rely on online visual information to guide our hand to intercept it. Updating predictions with online information allow to correct for any mismatch between the predicted target position and the hand position during an on-going movement, but any perceptual error that is still present at the last moment at which we can update our prediction will result in an equivalent interception error.

Somatosensory predictions in reaching

Speaker: Katja Fiehler, Department of Psychology and Sports Science, Giessen University, Giessen, Germany

Movement planning and execution lead to changes in somatosensory perception. For example, tactile stimuli on a moving compared to a resting limb are typically perceived as weaker and later in time. This phenomenon is termed tactile suppression and has been linked to a forward model mechanism which predicts the sensory consequences of the self-generated action and as a result discounts the respective sensory reafferences. As tactile suppression is also evident in passive hand movements, both predictive and postdictive mechanisms may be involved. However, its functional role is still widely unknown. It has been proposed that tactile suppression prevents sensory overload due to the large amount of afferent information generated during movement and therefore facilitates processing of external sensory events. However, if tactile feedback from the moving limb is needed to gain information, e.g. at the fingers involved in grasping, tactile sensitivity is less strongly reduced. In the talk, I will present recent results from a series of psychophysical experiments that show that tactile sensitivity is dynamically modulated during the course of the reaching movement depending on the reach goal and the predicted movement consequences. These results provide first evidence that tactile suppression may indeed free capacities to process other, movement-relevant somatosensory signals. Moreover, the observed perceptual changes were associated with adjustments in the motor system suggesting a close coupling of predictive mechanisms in perception and action.

Prediction during self-motion: the primate cerebellum selectively encodes unexpected vestibular information

Speaker: Kathleen Cullen, Department of Physiology, McGill University, Montréal, Québec, Canada

A prevailing view is that the cerebellum is the site of a forward model that predicts the expected sensory consequences of self-generated action. Changes in motor apparatus and/or environment will cause a mismatch between the cerebellum’s prediction and the actual resulting sensory stimulation. This mismatch – the ‘sensory prediction error,’ – is thought to be vital for updating both the forward model and motor program during motor learning to ensure that sensory-motor pathways remain calibrated. However, where and how the brain compares expected and actual sensory feedback was unknown. In this talk, I will first review experiments that focused on a relatively simple sensory-motor pathway with a well-described organization to gain insight into the computations that drive motor learning. Specifically, the most medial of the deep cerebellar nuclei (rostral fastigial nucleus), constitutes a major output target of the cerebellar cortex and in turn sends strong projections to the vestibular nuclei, reticular formation, and spinal cord to generate reflexes that ensure accurate posture and balance. Trial by trial analysis of these neurons in a motor learning task revealed the output of a computation in which the brain selectively encodes unexpected self-motion (vestibular information). This selectively enables both the i) rapid suppression of descending reflexive commands during voluntary movements and ii) rapid updating of motor programs in the face of changes to either the motor apparatus or external environment. I will then consider the implications of these findings regarding our recent work on the thalamo-cortical processing of vestibular information.

< Back to 2018 Symposia

Advances in temporal models of human visual cortex

Time/Room: Friday, May 18, 2018, 5:00 – 7:00 pm, Talk Room 2
Organizer(s): Jonathan Winawer, Department of Psychology and Center for Neural Science, New York University. New York, NY
Presenters: Geoffrey K. Aguirre, Christopher J. Honey, Anthony Stigliani, Jingyang Zhou

< Back to 2018 Symposia

Symposium Description

The nervous system extracts meaning from the distribution of light over space and time. Spatial vision has been a highly successful research area, and the spatial receptive field has served as a fundamental and unifying concept that spans perception, computation, and physiology. While there has also been a large interest in temporal vision, the temporal domain has lagged the spatial domain in terms of quantitative models of how signals are transformed across the visual hierarchy (with the notable exception of motion processing). In this symposium, we address the question of how multiple areas in human visual cortex encode information distributed over time. Several groups in recent years made important contributions to measuring and modeling temporal processing in human visual cortex. Some of this work shows parallels with spatial vision. For example, one important development has been the notion of a cortical hierarchy of increasingly long temporal windows, paralleling the hierarchy of spatial receptive fields (Hasson et al, 2009; Honey et al, 2012; Murray et al, 2014). A second type of study, from Geoff Aguirre’s lab, has combined the tradition of repetition suppression (Grill-Spector et al, 1999) with the notion of multiple time scales across the visual pathways to develop a computational model of how sequential stimuli are encoded in multiple visual areas (Mattar et al, 2016). Finally, several groups including the Grill-Spector lab and Winawer lab have extended the tools of population receptive field models from the spatial to the temporal domain, building models that predict how multiple cortical areas respond to arbitrary temporal sequences of visual stimulation (Horiguchi et al, 2009; Stigliani and Grill-Spector, 2017; Zhou et al 2017). Across the groups, there have been some common findings, such as the general tendency toward longer periods of temporal interactions in later visual areas. However, there are also a number of challenges in considering these recent developments together. For example, can (and should) we expect the same kind of theories and models to account for temporal interactions in both early visual areas at the time-scale of tens of milliseconds, and later visual areas at the time-scale of seconds or minutes? How do temporal properties of visual areas depend on spatial aspects of the stimuli? Should we expect principles of spatial computation, such as hierarchical pooling and normalization, to transfer analogously to the temporal domain? To what extent do temporal effects depend on task? Can temporal models at the scale of large neuronal populations (functional MRI, intracranial EEG) be explained in terms of the behavior of single neurons, and should this be a goal? Through this symposium, we aim to present an integrated view of the recent literature in temporal modeling of visual cortex, with each presenter both summarizing a recent topic and answering a common set of questions. The common questions posed to each presenter will be used to assess both the progress and the limits of recent work, with the goal of crystallizing where the field might go next in this important area.


Variation in Temporal Stimulus Integration Across Visual Cortex

Speaker: Geoffrey K. Aguirre, Department of Neurology, Perelman School of Medicine, University of Pennsylvania
Additional Authors: Marcelo G. Mattar, Princeton Neuroscience Institute, Princeton University; David A. Kahn, Department of Neuroscience, University of Pennsylvania; Sharon L. Thompson-Schill, Department of Psychology, University of Pennsylvania

Object percept is shaped by the long-term average of experience as well as immediate, comparative context. Measurements of brain activity have demonstrated corresponding neural mechanisms, including norm-based responses reflective of stored prototype representations, and adaptation induced by the immediately preceding stimulus. Our recent work examines the time-scale of integration of sensory information, and explicitly tests the idea that the apparently separate phenomena of norm-based coding and adaptation can arise from a single mechanism of sensory integration operating over varying timescales. We used functional MRI to measure neural responses from the fusiform gyrus while subjects observed a rapid stream of face stimuli. Neural activity at this cortical site was best explained by the integration of sensory experience over multiple sequential stimuli, following a decaying-exponential weighting function. While this neural activity could be mistaken for immediate neural adaptation or long-term, norm-based responses, it in fact reflected a timescale of integration intermediate to both. We then examined the timescale of sensory integration across the cortex. We found a gradient that ranged from rapid sensory integration in early visual areas, to long-term, stable representations towards higher-level, ventral-temporal cortex. These findings were replicated with a new set of face stimuli and subjects. Our results suggest that a cascade of visual areas integrate sensory experience, transforming highly adaptable responses at early stages to stable representations at higher levels.

Temporal Hierarchies in Human Cerebral Cortex

Speaker: Christopher J. Honey, Department of Psychological & Brain Sciences, Johns Hopkins University
Additional Authors: Hsiang-Yun Sherry Chien, Psychological and Brain Sciences, Johns Hopkins University; Kevin Himberger, Psychological and Brain Sciences, Johns Hopkins University

Our understanding of each moment of the visual world depends on the previous moment. We make use of temporal context to segregate objects, to accumulate visual evidence, to comprehend sequences of events, and to generate predictions. Temporal integration — the process of combining past and present information — appears not to be restricted to specialized subregions of the brain, but is widely distributed across the cerebral cortex. In addition, temporal integration processes appear to be systematically organized into a hierarchy, with gradually greater context dependence as one moves toward higher order regions. What is the mechanistic basis of this temporal hierarchy? What are its implications for perception and learning, especially in determining the boundaries between visual events? How does temporal integration relate to the processes supporting working memory and episodic memory? After reviewing the evidence around each of these questions, I will describe a computational model of hierarchical temporal processing in the human cerebral cortex. Finally, I will describe our tests of the predictions of this model for for brain and behavior, in settings where where humans perceive and learn nested temporal structure.

Modeling the temporal dynamics of high-level visual cortex

Speaker: Anthony Stigliani, Department of Psychology, Stanford University
Additional Authors: Brianna Jeska, Department of Psychology, Stanford University; Kalanit Grill-Spector, Department of Psychology, Stanford University

How is temporal information processed in high-level visual cortex? To address this question, we measured cortical responses with fMRI (N = 12) to time-varying stimuli across 3 experiments using stimuli that were either transient, sustained, or contained both transient and sustained stimulation and ranged in duration from 33ms to 20s. Then we implemented a novel temporal encoding model to test how different temporal channels contribute to responses in high-level visual cortex. Different than the standard linear model, which predicts responses directly from the stimulus, the encoding approach first predicts neural responses to the stimulus with fine temporal precision and then derives fMRI responses from these neural predictions. Results show that an encoding model not only explains responses to time varying stimuli in face- and body-selective regions, but also finds differential temporal processing across high-level visual cortex. That is, we discovered that temporal processing differs both across anatomical locations as well as across regions that process different domains. Specifically, face- and body-selective regions in lateral temporal cortex (LTC) are dominated by transient responses, but face- and body-selective regions in lateral occipital cortex (LOC) and ventral temporal cortex (VTC) illustrate both sustained and transient responses. Additionally, the contribution of transient channels in body-selective regions is higher than in neighboring face-selective regions. Together, these results suggest that domain-specific regions are organized in parallel processing streams with differential temporal characteristics and provide evidence that the human visual system contains a separate lateral processing stream that is attuned to changing aspects of the visual input.

Dynamics of temporal summation in human visual cortex

Speaker: Jingyang Zhou, Department of Psychology, New York University
Additional Authors: Noah C. Benson, Psychology, New York University; Kendrick N. Kay, Center for Magnetic Resonance Research, Department of Radiology, University of Minnesota, Twin Cities; Jonathan Winawer, Psychology and Center for Neural Science, New York University

Later visual areas become increasingly tolerant to variations in image properties such as object size, location, viewpoint, and so on. This phenomenon is often modeled by a cascade of repeated processing stages in which each stage involves pooling followed by a compressive nonlinearity. One result of this sequence is that stimulus-referred measurements show increasingly large receptive fields and stronger normalization. Here, we apply a similar approach to the temporal domain. Using fMRI and intracranial potentials (ECoG), we develop a population receptive field (pRF) model for temporal sequences of visual stimulation. The model consists of linear summation followed by a time-varying divisive normalization. The same model accurately accounts for both ECoG broadband time course and fMRI amplitudes. The model parameters reveal several regularites about temporal encoding in cortex. First, higher visual areas accumulate stimulus information over a longer time period than earlier areas, analogous to the hierarchically organized spatial receptive fields. Second, we found that all visual areas sum sub-linearly in time: e.g., the response to a long stimulus is less than the response to two successive brief stimuli. Third, the degree of compression increases in later visual areas, analogous to spatial vision. Finally, based on published data, we show that our model can account for the time course of single units in macaque V1 and multiunits in humans. This indicates that for space and time, cortex uses a similar processing strategy to achieve higher-level and increasingly invariant representations of the visual world.

< Back to 2018 Symposia

When seeing becomes knowing: Memory in the form perception pathway

Time/Room: Friday, May 18, 2018, 2:30 – 4:30 pm, Talk Room 2
Organizer(s): Caitlin Mullin, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of the Technology
Presenters: Wilma Bainbridge, Timothy Brady, Gabriel Kreiman, Nicole Rust, Morgan Barense, Nicholas Turk-Browne

< Back to 2018 Symposia

Symposium Description

Classic accounts of how the brain sees and remembers largely describes vision and memory as distinct systems, where information about the content of a scene is processed in the ventral visual stream (VVS) and our memories of scenes past are processed by independent structures in the Medial Temporal Lobe (MTL). However, more recent work has begun to challenge this view by demonstrating interactions and dependencies between visual perception and memory at nearly every stage of the visual processing hierarchy. In this symposium, we will present a series of cutting edge behavioural and neuroscience studies that showcase an array of crossmethodological approaches (psychophysics, fMRI, MEG, single unit recording in monkeys, human E-CoG) to establish that perception and memory are part of a shared, bidirectional, interactive network. Our symposium will begin with Caitlin Mullin providing an overview of the contemporary problems associated with the traditional memory/perception framework. Next, Wilma Bainbridge will describe the factors that give rise to image memorability. Tim Brady will follow with a description of how the limits of encoding affect visual memory storage and retrieval. Gabriel Kreiman will focus on how our brains interpret visual images that we have never encountered before by drawing on memory systems. Nicole Rust will present evidence that one of the same VVS brain areas implicated in visual object recognition, monkey IT cortex, also reflects visual memory signals that are well-aligned with behavioral reports of remembering and forgetting. Morgan Barense will describe the transformation between the neural coding of low level perceptual to high level conceptual features in one brain area that lies within the MTL, perirhinal cortex. Finally, Nick Turk-Browne will describe the role of the hippocampus in generating expectations that work in a top-down manner to influence our perceptions. Our symposium will culminate with a discussion focused on how we can develop an integrative framework that provides a full account of the interactions between vision and memory, including extending state-of-the art computational models of visual processing to also incorporate visual memory, as well as understanding how dysfunction in the interactions between vision and memory systems lead to memory disorders. The findings and resulting discussions presented in this symposium will be targeted broadly and will reveal important considerations for anyone, at any level of their career (student, postdoc or faculty), interested in the interactions between visual perception and memory.


Memorability – predicting memory from visual information, and measuring visual information from memory

Speaker: Wilma Bainbridge, National Institute of Mental Health

While much of memory research focuses on the memory behavior of individual participants, little memory work has looked at the visual attributes of the stimulus that influence future memory. However, in recent work, we have found that there are surprising consistencies to the images people remember and forget, and that the stimulus ultimately plays a large part in predicting later memory behavior. This consistency in performance can then be measured as a perceptual property of any stimulus, which we call memorability. Memorability can be easily measured in the stimuli of any experiment, and thus can be used to determine the degree previously found effects could be explained by the stimulus. I will present an example where we find separate neural patterns sensitive to stimulus memorability and individual memory performance, through re-analyzing the data and stimuli from a previously published fMRI memory retrieval experiment (Rissman et al., 2010). I will also show how memorability can be easily taken into account when designing experiments to ask fundamental questions about memory, such as – are there differences between the types of images people can recognize versus the types of images people can recall? I will present ways for experimenters to easily measure or control for memorability in their own experiments, and also some new ways quantify the visual information existing within a memory.

The impact of perceptual encoding on subsequent visual memory

Speaker: Timothy Brady, University of California San Diego

Memory systems are traditionally associated with the end stages of the visual processing sequence: attending to a perceived object allows for object recognition; information about this recognized object is stored in working memory; and eventually this information is encoded into an abstract long-term memory representation. In this talk, I will argue that memories are not truly abstract from perception: perceptual distinctions persist in memory, and our memories are impacted by the perceptual processing that is used to create them. In particular, I will talk about evidence that suggests that both visual working memory and visual long-term memory are limited by the quality and nature of their perceptual encoding, both in terms of the precision of the memories that are formed and their structure.

Rapid learning of meaningful image interpretation

Speaker: Gabriel Kreiman, Harvard University

A single event of visual exposure to new information may be sufficient for interpreting and remembering an image. This rapid form of visual learning stands in stark contrast with modern state-of-the-art deep convolutional networks for vision. Such models thrive in object classification after supervised learning with a large number of training examples. The neural mechanisms subserving rapid visual learning remain largely unknown. I will discuss efforts towards unraveling the neural circuits involved in rapid learning of meaningful image interpretation in the human brain. We studied single neuron responses in human epilepsy patients to instances of single shot learning using Mooney images. Mooney images render objects in binary black and white in such a way that they can be difficult to recognize. After exposure to the corresponding grayscale image (and without any type of supervision), it becomes easier to recognize the objects in the original Mooney image. We will demonstrate a single unit signature of rapid learning in the human medial temporal lobe and provide initial steps to understand the mechanisms by which top-down inputs can rapidly orchestrate plastic changes in neuronal circuitry.

Beyond identification: how your brain signals whether you’ve seen it before

Speaker: Nicole Rust, University of Pennsylvania

Our visual memory percepts of whether we have encountered specific objects or scenes before are hypothesized to manifest as decrements in neural responses in inferotemporal cortex (IT) with stimulus repetition. To evaluate this proposal, we recorded IT neural responses as two monkeys performed variants of a single-exposure visual memory task designed to measure the rates of forgetting with time and the robustness of visual memory to a stimulus parameter known to also impact IT firing rates, image contrast. We found that a strict interpretation of the repetition suppression hypothesis could not account for the monkeys’ behavior, however, a weighted linear read-out of the IT population response accurately predicted forgetting rates, reaction time patterns, individual differences in task performance and contrast invariance. Additionally, the linear weights were largely all the same-sign and consistent with repetition suppression. These results suggest that behaviorally-relevant memory information is in fact reflected in via repetition suppression in IT, but only within an IT subpopulation.

Understanding what we see: Integration of memory and perception in the ventral visual stream

Speaker: Morgan Barense, University of Toronto

A central assumption in most modern theories of memory is that memory and perception are functionally and anatomically segregated. For example, amnesia resulting from medial temporal lobe (MTL) lesions is traditionally considered to be a selective deficit in long-term declarative memory with no effect on perceptual processes. The work I will present offers a new perspective that supports the notion that memory and perception are inextricably intertwined, relying on shared neural representations and computational mechanisms. Specifically, we addressed this issue by comparing the neural pattern similarities among object-evoked fMRI responses with behavior-based models that independently captured the visual and conceptual similarities among these stimuli. Our results revealed evidence for distinctive coding of visual features in lateral occipital cortex, and conceptual features in the temporal pole and parahippocampal cortex. By contrast, we found evidence for integrative coding of visual and conceptual object features in the perirhinal cortex of the MTL. Taken together, our findings suggest that perirhinal cortex uniquely supports the representation of fully-specified object concepts through the integration of their visual and conceptual features.

Hippocampal contributions to visual learning

Speaker: Nicholas Turk-Browne, Yale University

Although the hippocampus is usually viewed as a dedicated memory system, its placement at the top of, and strong interactions with, the ventral visual pathway (and other sensory systems) suggest that it may play a role in perception. My lab has recently suggested one potential perceptual function of the hippocampus — to learn about regularities in the environment and then to generate expectations based on these regularities that get reinstated in visual cortex to influence processing. I will talk about several of our studies using high-resolution fMRI and multivariate methods to characterize such learning and prediction.

< Back to 2018 Symposia

Vision and Visualization: Inspiring Novel Research Directions in Vision Science

Time/Room: Friday, May 18, 2018, 12:00 – 2:00 pm, Talk Room 2
Organizer(s): Christie Nothelfer, Northwestern University; Madison Elliott, UBC, Zoya Bylinskii, MIT, Cindy Xiong, Northwestern University, & Danielle Albers Szafir, University of Colorado Boulder
Presenters: Ronald A. Rensink, Aude Oliva, Steven Franconeri, Danielle Albers Szafir

< Back to 2018 Symposia

Symposium Description

Data is ubiquitous in the modern world, and its communication, analysis, and interpretation are critical scientific issues. Visualizations leverage the capabilities of the visual system, allowing us to intuitively explore and generate novel understandings of data in ways that fully-automated approaches cannot. Visualization research builds an empirical framework around design guidelines, perceptual evaluation of design techniques, and a basic understanding of the visual processes associated with viewing data displays. Vision science offers the methodologies and phenomena that can provide foundational insight into these questions. Challenges in visualization map directly to many vision science topics, such as finding data of interest (visual search), estimating data means and variance (ensemble coding), and determining optimal display properties (crowding, salience, color perception). Given the growing interest in psychological work that advances basic knowledge and allows for immediate translation, visualization provides an exciting new context for vision scientists to confirm existing hypotheses and explore new questions. This symposium will illustrate how interdisciplinary work across vision science and visualization simultaneously improves visualization techniques while advancing our understanding of the visual system, and inspire new research opportunities at the intersection of these two fields.

Historically, the crossover between visualization and vision science relied heavily on canonical findings, but this has changed significantly in recent years. Visualization work has recently incorporated and iterated on newer vision research, and the results has been met with great excitement from both sides (e.g., Rensink & Baldridge, 2010; Haroz & Whitney, 2012; Harrison et al., 2014; Borkin et al., 2016; Szafir et al., 2016). Unfortunately, very little of this work is presented regularly at VSS, and there is currently no dedicated venue for collaborative exchanges between the two research communities. This symposium showcases the current state of vision science and visualization research integration, and aspires to make VSS a home for future exchanges. Visualization would benefit from sampling a wider set of vision topics and methods, while vision scientists would gain a new real-world context that simultaneously provokes insight about the visual system and holds translational impact.

This symposium will first introduce the benefits of collaboration between vision science and visualization communities, including the discussion of a specific example: correlation perception (Ronald Rensink). Next, we will discuss the properties of salience in visualizations (Aude Oliva), how we extract patterns, shapes, and relations from data points (Steven Franconeri), and how color perception is affected by the constraints of visualization design (Danielle Albers Szafir). Each talk will be 25 minutes long. The speakers, representing both fields, will demonstrate how studying these topics in visualizations has uniquely advanced our understanding of the visual system, as well as what research in these cross-disciplinary projects looks like, and propose open questions to propel new research in both communities. The symposium will conclude with an open discussion about how vision science and visualization communities can mutually benefit from deeper integration. We expect these topics to be of interest to VSS members from a multitude of vision science topics, specifically: pattern recognition, salience, shape perception, color perception, and ensemble coding.


Information Visualization and the Study of Visual Perception

Speaker: Ronald A. Rensink, Departments of Psychology and Computer Science, UBC

Information visualization and vision science can interact in three different (but compatible) ways. The first uses knowledge of human vision to design more effective visualizations. The second adapts measurement techniques originally developed for experiments to assess performance on given visualizations. And a third way has also been recently proposed: the study of restricted versions of existing visualizations. These can be considered as “fruit flies”, i.e., systems that exist in the real world, but are still simple enough to study. This approach can help us discover why a visualization works, and can give us new insights into visual perception as well. An example of this is the perception of Pearson correlation in scatterplots. Performance here can be described by two linked laws: a linear one for discrimination and a logarithmic one for perceived magnitude (Rensink & Baldridge, 2010). These laws hold under a variety of conditions, including when properties other than spatial position are used to convey information (Rensink, 2014). Such behavior suggests that observers can infer probability distributions in an abstract two-dimensional parameter space (likely via ensemble coding), and can use these to estimate entropy (Rensink, 2017). These results show that interesting aspects of visual perception can be discovered using restricted versions of real visualization systems. It is argued that the perception of correlation in scatterplots is far from unique in this regard; a considerable number of these “fruit flies” exist, many of which are likely to cast new light on the intelligence of visual perception.

Where do people look on data visualizations?

Speaker: Aude Oliva, Massachusetts Institute of Technology
Additional Authors: Zoya Bylinskii, MIT

What guides a viewer’s attention when she catches a glimpse of a data visualization? What happens when the viewer studies the visualization more carefully, to complete a cognitively-demanding task? In this talk, I will discuss the limitations of computational saliency models for predicting eye fixations on data visualizations (Bylinskii et al., 2017). I will present perception and cognition experiments to measure where people look in visualizations during encoding to, and retrieval from, memory (Borkin, Bylinskii, et al., 2016). Motivated by clues that eye fixations give about higher-level cognitive processes like memory, we sought a way to crowdsource attention patterns at scale. I will introduce BubbleView, our mouse-contingent interface to approximate eye tracking (Kim, Bylinskii, et al., 2017). BubbleView presents participants with blurred visualizations and allows them to click to expose “bubble” regions at full resolution. We show that up to 90% of eye fixations on data visualizations can be accounted for by the BubbleView clicks of online participants completing a description task. Armed with a tool to efficiently and cheaply collect attention patterns on images, which we call “image importance” to distinguish from “saliency”, we collected BubbleView clicks for thousands of visualizations and graphic designs to train computational models (Bylinskii et al., 2017). Our models run in real-time to predict image importance on new images. This talk will demonstrate that our models of attention for natural images do not transfer to data visualizations, and that using data visualizations as stimuli for perception studies can open up fruitful new research directions.

Segmentation, structure, and shape perception in data visualizations

Speaker: Steven Franconeri, Northwestern University

The human visual system evolved and develops to perceive scenes, faces, and objects in the natural world, and this is where vision scientists justly focus their research. But humans have adapted that system to process artificial worlds on paper and screens, including data visualizations. I’ll demonstrate two examples of how studying the visual system within such worlds can provide vital cross-pollination for our basic research. First a complex line or bar graph can be alternatively powerful, or vexing, for students and scientists. What is the suite of our available tools for extracting the patterns within it? Our existing research is a great start: I’ll show how the commonly encountered ‘magical number 4’ (Choo & Franconeri, 2013) limits processing capacity, and how the literature on shape silhouette perception could predict how we segment them. But even more questions are raised: what is our internal representation of the ‘shape’ of data – what types of changes to the data can we notice, and what changes would leave us blind? Second, artificial displays require that we recognize relationships among objects (Lovett & Franconeri, 2017), as when you quickly extract two main effects and an interaction from a 2×2 bar graph. We can begin to explain these feats through multifocal attention or ensemble processing, but soon fall short. I will show how these real-world tasks inspire new research on relational perception, highlighting eyetracking work that reveals multiple visual tools for extracting relations based on global shape vs. contrasts between separate objects.

Color Perception in Data Visualizations

Speaker: Danielle Albers Szafir, University of Colorado Boulder

Many data visualizations use color to convey values. These visualizations commonly rely on vision science research to match important properties of data to colors, ensuring that people can, for example, identify differences between values, select data subsets, or match values against a legend. Applying vision research to color mappings also creates new questions for vision science. In this talk, I will discuss several studies that address knowledge gaps in color perception raised through visualization, focusing on color appearance, lightness constancy, and ensemble coding. First, conventional color appearance models assume colors are applied to 2° or 10° uniformly-shaped patches; however, visualizations map colors to small shapes (often less than 0.5°) that vary in their size and geometry (e.g., bar graphs, line charts, or maps), degrading difference perceptions inversely with a shape’s geometric properties (Szafir, 2018). Second, many 3D visualizations embed data along surfaces where shadows may obscure data, requiring lightness constancy to accurately resolve values. Synthetic rendering techniques used to improve interaction or emphasize aspects of surface structure manipulate constancy, influencing people’s abilities to interpret shadowed colors (Szafir, Sarikaya, & Gleicher, 2016). Finally, visualizations frequently require ensemble coding of large collections of values (Szafir et al., 2016). Accuracy differences between different visualizations for value identification (e.g., extrema) and summary tasks (e.g., mean) suggest differences in ensemble processing for color and position (Albers, Correll, & Gleicher, 2014). I will close by discussing open challenges for color perception arising from visualization design, use, and interpretation.

< Back to 2018 Symposia

2018 Symposia

Clinical insights into basic visual processes

Organizer(s): Paul Gamlin, University of Alabama at Birmingham; Ann E. Elsner, Indiana University; Ronald Gregg, University of Louisville
Time/Room: Friday, May 18, 2018, 12:00 – 2:00 pm, Talk Room 1

This year’s biennial ARVO at VSS symposium features insights into human visual processing at the retinal and cortical level arising from clinical and translational research. The speakers will present recent work based on a wide range of state-of-the art techniques including adaptive optics, brain and retinal imaging, psychophysics and gene therapy. More…

Vision and Visualization: Inspiring novel research directions in vision science

Organizer(s): Christie Nothelfer, Northwestern University; Madison Elliott, UBC, Zoya Bylinskii, MIT, Cindy Xiong, Northwestern University, & Danielle Albers Szafir, University of Colorado Boulder
Time/Room: Friday, May 18, 2018, 12:00 – 2:00 pm, Talk Room 2

Visualization research seeks design guidelines for efficient visual displays of data. Vision science topics, such as pattern recognition, salience, shape perception, and color perception, all map directly to challenges encountered in visualization, raising new vision science questions and creating a space ripe for collaboration. Four speakers representing both vision science and visualization will discuss recent cross-disciplinary research, closing with a panel to discuss about how vision science and visualization communities can mutually benefit from deeper integration. This symposium will demonstrate that contextualizing vision science research in visualization can expose novel gaps in our knowledge of how perception and attention work. More…

Prediction in perception and action

Organizer(s): Katja Fiehler, Department of Psychology and Sports Science, Giessen University, Giessen, Germany
Time/Room: Friday, May 18, 2018, 2:30 – 4:30 pm, Talk Room 1

Prediction is an essential mechanism enabling humans to prepare for future events. This is especially important in a dynamically changing world, which requires rapid and accurate responses to external stimuli. While it is unquestionable that predictions play a fundamental role in perception and action, their underlying mechanisms and neural basis are still poorly understood. The goal of this symposium is to integrate recent findings from psychophysics, sensorimotor control, and electrophysiology to provide a novel and comprehensive view on predictive mechanisms in perception and action spanning from behavior to neurons and from strictly laboratory tasks to (virtual) real world scenarios. More…

When seeing becomes knowing: Memory in the form perception pathway

Organizer(s): Caitlin Mullin, Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of the Technology
Time/Room: Friday, May 18, 2018, 2:30 – 4:30 pm, Talk Room 2

The established view of perception and memory is that they are dissociable processes that recruit distinct brain structures, with visual perception focused on the ventral visual stream and memory subserved by independent deep structures in the medial temporal lobe. Recent work in cognitive neuroscience has challenged this traditional view by demonstrating interactions and dependencies between perception and memory at nearly every stage of the visual hierarchy. In this symposium, we will present a series of cutting edge studies that showcase cross-methodological approaches to describe how visual perception and memory interact as part of a shared, bidirectional, interactive network. More…

Visual remapping: From behavior to neurons through computation

Organizer(s): James Mazer, Cell Biology & Neuroscience, Montana State University, Bozeman, MT & Fred Hamker, Chemnitz University of Technology, Chemnitz, Germany
Time/Room: Friday, May 18, 2018, 5:00 – 7:00 pm, Talk Room 1

In this symposium we will discuss the neural substrates responsible for maintaining stable visual and attentional representations during active vision. Speakers from three complementary experimental disciplines, psychophysics, neurophysiology and computational modeling, will discuss recent advances in clarifying the role of spatial receptive field “remapping” in stablizing sensory representations across saccadic eye movements. Participants will address new experimental and theoretical methods for characterizing statiotemporal dynamics of visual and attentional remapping, both behavioral and physiological, during active vision and relate these data to recent computational efforts towards modeling oculomotor and visual system interactions. More…

Advances in temporal models of human visual cortex

Organizer(s): Jonathan Winawer, Department of Psychology and Center for Neural Science, New York University. New York, NY
Time/Room: Friday, May 18, 2018, 5:00 – 7:00 pm, Talk Room 2

How do multiple areas in the human visual cortex encode information distributed over time? We focus on recent advances in modeling the temporal dynamics in the human brain: First, cortical areas have been found to be organized in a temporal hierarchy, with increasingly long temporal windows from earlier to later visual areas. Second, responses in multiple areas can be accurately predicted with temporal population receptive field models. Third, quantitative models have been developed to predict how responses in different visual areas are affected by both the timing and content of the stimulus history (adaptation). More…

Vision Sciences Society