Perception and Action: Reach, grasp, walk
Talk Session: Sunday, May 21, 2023, 10:45 am – 12:30 pm, Talk Room 2
Moderator: William Warren, Brown University
Schedule of Events | Search Abstracts | Symposia | Talk Sessions | Poster Sessions | Abstract Numbering
Talk 1, 10:45 am, 32.21
Decoding Features of Real-world Navigation from the Human Hippocampus
Kathryn N. Graves1 (), Ariadne Letrou1, Tyler E. Gray2, Imran H. Quraishi2, Nicholas B. Turk-Browne1,3; 1Department of Psychology, Yale University, 2Department of Neurology, Yale University, 3Wu Tsai Institute, Yale University
Neuronal recordings during animal navigation and virtual human navigation have revealed specialized spatial coding properties in the hippocampus for representing an organism's location, speed, and direction. However, it remains unclear how these representations operate during human navigation in the real world and how they manifest at the level of local field potentials (LFPs). Here we report evidence of stable real-world navigation coding in the human hippocampus. Epilepsy patients chronically implanted with a Responsive Neurostimulation (RNS) device walked back and forth along a linear track while hippocampal LFP data were recorded from two or four channels. Position along the track was measured continuously using an optical head tracker, from which we could label neural data with instantaneous heading direction, speed, and location. We trained three models for each channel and patient to predict these labels from hippocampal LFPs over time: a binary classifier to decode which of the two heading directions and two regression models to predict continuous estimates of speed and location, respectively. These models successfully predicted direction, speed, and location more accurately than chance. The channels that represented these variables overlapped significantly, some coding for two and others for all three. Finally, we investigated the stability of these representations in a subset of repeatedly tested patients by training models on their initial session and testing on subsequent sessions. These across-session models significantly predicted speed and location, but not direction, and the channels representing speed overlapped across sessions. These results demonstrate that complex features of human movement through space are represented in neuronal population activity in the hippocampus. The innovative RNS platform we developed allows for highly precise electrophysiological measurements in the hippocampus during real-world activities, creating untold possibilities for investigating neural mechanisms of navigation, memory, and other hippocampal-dependent behaviors in ecologically valid tasks and settings.
Acknowledgements: This work was supported by National Institutes of Health (NIH) Grant R01MH069456 (to N. B. Turk-Browne), a Swebilius Foundation Grant (to I. H. Quraishi), an NIH Grant 1F99NS125835-01 (to K. N. Graves), and a Swebilius Foundation Grant (to K. N. Graves)
Talk 2, 11:00 am, 32.22
Dynamic collision envelope in virtual reality walking with colliding pedestrians
Jae-Hyun Jung1 (), Alex Hwang2, Jonathan Doyon3, Sujin Kim4; 1Department of Ophthalmology, Harvard Medical School, 2Schepens Eye Research Institute of Massachusetts Eye and Ear
When walking, we detect possible collisions with other pedestrians and avoid them, estimating body volumes and safety margins. This safety margin, collision envelope, has been measured in limited conditions: only parallel approaching collisions with lateral offset were tested while subjects watched a walking video. These scenarios limit generalizability for real-world walking where walkers navigate freely with various approaching directions of pedestrians. To evaluate realistic and dynamic collision envelopes in a risk-free environment, we developed a virtual reality walking scenario using the Meta Quest 2 head-mounted display (HMD). While a subject walking with gaze movement in an empty real-world corridor, a corresponding virtual shopping mall with pedestrians approached from 20°, 40°, or 60° bearing angles on a collision course face-to-face or overtaken were shown on HMD. 10 non-colliding pedestrians on various walking paths were also present. Subjects were asked to freely and naturally avoided potential collisions (walking path or speed change). Subjects with homonymous hemianopia (HH; n=6) and subjects with normal vision (NV; n=8) avoided 20 face-to-face and 20 overtaken pedestrians. As a result of the collision avoidance behavior, the trajectories of pedestrians relative to the subjects were changed, and thus the safety margins in various paths were collected. Dynamic collision envelope was calculated as the area kept as the safety margin in more than 50% of trials. HH subjects had larger envelopes (0.95m2, SD=0.60) than NV subjects (0.71m2, SD=0.42; p=0.044) and envelopes were larger when colliders were approaching (1.14m2, SD=0.53) compared to overtaken (0.49, SD=0.19; p<0.001). These results may suggest a more conservative safety margin in HH than NV when avoiding potential collisions. Since the relative walking speeds of the approaching pedestrians were faster than the overtaken pedestrians, estimated time-to-collision may also affect the size and structure of the collision envelope.
Acknowledgements: Supported in part by a NIH grant R01-EY031777
Talk 3, 11:15 am, 32.23
Motion Energy Modulates Feature Tracking in Human Locomotor Control
Zhenyu Zhu1 (), William H. Warren1; 1Brown University
Several visual variables are known to influence motion perception, including motion energy (1st-order motion) and tracking the positions of features (3rd-order motion) (Lu and Sperling 1995), but their contribution to locomotor control is unknown. Previously, we found that 3rd-order motion dominated locomotor responses in a crowd following task. Here, we manipulate the strength of 1st- and 3rd-order motion and measure their effects on heading and speed responses during crowd following. Human participants wore a head-mounted display (101ºH x 105ºV, 90 Hz) and were asked to “walk with” nine virtual objects textured with Julesz pattern in a mid-gray environment. The objects initially moved forward at 1.2 m/s, then their heading direction (±20º) or speed (± 0.2m/s) was perturbed for 2s. During the perturbation, the texture within object boundaries was rendered with a 4-stroke reverse-phi illusion, such that it moved laterally (heading perturbation) or radially (speed perturbation). The direction of the 1st-order texture motion and the 3rd-order boundary motion were either the Same or Opposite direction. Locomotor responses were always in the same direction as object boundaries. When both texture and boundaries are clearly defined (Experiment 1), heading responses were actually larger in the Opposite condition than the Same condition (p<0.05), suggesting a motion-contrast effect. When the strength of the boundary feature was reduced by blurring with a Gaussian filter (Experiment 2), both heading and speed responses were reduced (p<0.001), revealing an influence of motion energy. When the strength of motion energy was reduced by blurring the texture (Experiment 3), the motion contrast effect in the heading condition was eliminated. The results indicate that human locomotor responses are dominated by (3rd-order) tracking of object features, but are slightly modulated by (1st-order) motion energy, depending on their relative strength. This suggests that 1st- and 3rd-order motion signals are integrated prior to locomotor control.
Acknowledgements: Funding: NIH R01EY029745
Talk 4, 11:30 am, 32.24
Visual detection while walking: Sensitivity modulates over the gait cycle
Cameron Phan1 (), David Alais1, Frans Verstraten1, Matthew Davidson1; 1University of Sydney
We investigated visual sensitivity over the gait cycle for stimuli presented at two eccentricities. Participants (n = 33) wore a virtual reality headset and walked along a smooth, level path while making a trigger-pull response to indicate if a briefly flashed visual target was detected. The small ellipsoid target was varied in contrast against a grey background to determine detection thresholds. Thresholds were measured at eccentricities of 4° or 12° from a central fixation cross, and while standing still or walking at natural speed. There were 190 stimuli per condition, presented at various jittered time points along the path. Head position data were used to divide the walking sequence into individual steps so that the data could be pooled into a single densely sampled gait cycle. Performance modulated over the gait cycle with an approximate sinusoidal pattern and fitted Fourier functions for response accuracy and response time revealed a modulation rate of 3.21 Hz for both variables. Accuracy modulated in-phase with gait cycle but was phase-shifted for response time. Overall, contrast thresholds were higher for peripheral targets regardless of motion condition. These results uncover the effect of walking on visual detection ability and its interaction with visual eccentricity. What would its effect be on higher visual abilities? Would other visual phenomena have different interactions?
Talk 5, 11:45 am, 32.25
MEG signatures of arm posture coding and integration into movement plans
Gunnar Blohm1 (), Doug Cheyne2, Doug Crawford3; 1Queen's University, Centre for Neuroscience Studies, VISTA, CAPnet, 2University of Toronto, The Hospital for Sick Children Research Institute, 3York University, Centre for Vision Research, VISTA, CAPnet
To plan a visually guided movement, the brain must calculate an extrinsic movement vector and then convert this into intrinsic muscle commands for current posture. Where, how and when this happens in the human cortical arm movement planning network remains largely unknown. Here, we use high spatiotemporal resolution magnetoencephalography (MEG) combined with a pro-/anti-wrist pointing task with 3 different forearm postures to investigate this question. First, we then computed cortical source activity in 16 previously identified bilateral cortical areas (Alikhanian, et al., Frontiers in Neuroscience 2013). We then compared pro/anti trials to identify brain areas coding for stimulus direction vs. movement direction. Sensory activity in α / β bands progressed from posterior to anterior cortical areas, culminating in a β-band movement plan in primary motor cortex. During the delay, movement codes then retroactively replaced the sensory code in more posterior areas (Blohm, et al., Cerebral Cortex 2019). We then contrasted oscillatory activity related to opposing wrist postures to find posture coding and test when and where the extrinsic-to-intrinsic transformation occurred. We found a distinct pair of overlapping networks coding for posture (in γ band) vs. posture-specific movement plans (β). Together with previous results showing a yet again different sub-network specifying motor effector and it’s integration (Blohm, et al., J Neurophysiol 2022), we demonstrate that distinct cortical sub-networks carry out different spatiotemporal computations for movement planning.
Acknowledgements: CIHR (Canada), Marie Curie Fellowships (EU), NSERC (Canada)
Talk 6, 12:00 pm, 32.26
Sensorimotor adaptation reveals systematic biases of 3D estimates for reach-to-grasp actions.
Chaeeun Lim1 (), Fulvio Domini1; 1Brown University
Previous studies found a systematic bias in depth perception: providing more depth cues increases the magnitude of perceived depth. Here we asked whether this bias is an artifact of cognitively driven perceptual judgements or reflects hardwired mechanisms of 3D processing which should also affect visually guided actions (e.g., reach-to-grasp movements). In the latter case, we expect an intuitive result: adding depth cues to an object that is initially grasped accurately should yield a grasping error. Importantly, this error would trigger correction mechanisms unbeknownst to the subject, giving rise to sensorimotor adaptation. We tested this hypothesis in an experiment where subjects repeatedly reached for and grasped a rendered 3D gaussian bump while receiving independently controlled haptic feedback. In the first block of trials, the 3D bump was only visually specified by binocular disparity. The participants’ movements rapidly adapted to this stimulus until the grasp stabilized. In the second block, when we added monocular cues (texture and shading) to the display, we detected the unintuitive result: the planned grip aperture was initially overestimated and then rapidly diminished in subsequent trials. This clear evidence of sensorimotor adaptation was confirmed in the third block where removing the monocular cues elicited an aftereffect. In the fourth block, we reintroduced again the monocular cues but reduced the rendered depth, so that the (overestimated) visual stimulus perceptually matched the haptic stimulus. As expected, in spite of the visuo-haptic mismatch, we did not detect any sensorimotor adaptation. Strikingly, the results of this novel and implicit test show that biased depth estimates drive both perception and action.
Talk 7, 12:15 pm, 32.27
Human see, human do: comparing visual and motor representations of hand gestures
Hunter Schone1,2 (), Tamar Makin2,3, Chris Baker1; 1Laboratory of Brain & Cognition, National Institute of Mental Health, National Institutes of Health, 2Institute of Cognitive Neuroscience, University College London, 3MRC Cognition and Brain Sciences Unit, University of Cambridge
Our hands are the primary means for interacting with our surroundings. As such, they are supported by a plethora of relevant representations in the brain, most notable are the somatosensory and motor representations in sensorimotor cortex and visual representations within occipitotemporal cortex, respectively. Here, we compared the representational structure when observing and executing hand gestures within and across visual and sensorimotor cortices using 3T functional MRI and 8-channel electromyography in human participants (n=60). To characterize both visual and motor features of hand representation, participants performed a visuomotor task that required them to either execute a specific hand gesture (8 gestures: open, close, pinch, tripod, one finger, two finger, three fingers, four fingers) or to observe a first-person video of a biological or robotic hand perform the same gesture. First, when visualizing the univariate activity for the contrast: actions vs. observations, we found an expected preference for actions in sensorimotor cortex. However, within OTC, we found separate regions that prefer hand actions (anterior portion) and hand observation (posterior portion). Next, using representational similarity analysis, we quantified the multivariate representational structure of observed and executed hand gestures in both regions. We found that OTC has more separable representations for hand actions and observations compared to sensorimotor cortex. When quantifying just observations, we found distinct representational structure between observations within OTC only, which were similar for observations of biological or robotic hands, and limited similarity in the structure of observations between regions. Surprisingly, when quantifying just hand actions, we found similar distances between actions in OTC as sensorimotor cortex and a strong correlation when comparing the representational structure between the two regions. Collectively, these results reveal fine-grained representational structure about hands in OTC that is similar for both observation and action and suggest a systematic visuomotor organization within OTC.