Lesioning category-selective units in silico yields functionally specialized deficits

Poster Presentation 56.423: Tuesday, May 23, 2023, 2:45 – 6:45 pm, Pavilion
Session: Object Recognition: Neural mechanisms

Jacob S. Prince1 (), George A. Alvarez1, Talia Konkle1; 1Harvard University

Hallmark neuropsychological findings show that lesions within the ventral visual stream yield dissociable visual recognition deficits for different categories. It has remained unclear how to reconcile these observations with distributed population coding theories, which suggest that neurons decode object category information from multivariate (high, low, and medium) activity patterns. Here, we clarify the relationship between these prominent theoretical frameworks by simulating the functional impact of lesions applied to category-selective units that emerge within self-supervised deep neural networks. Using fMRI-inspired localizer techniques, we identified groups of units with selectivity for faces, scenes, bodies, and words along the hierarchy of AlexNet trained with a domain-general instance discrimination objective (Barlow Twins). After fitting a linear classifier to assess the model’s ImageNet performance, we measured the category-level deficits that arose from lesioning each group of units. We found that the degree to which individual categories drove a strong response before lesioning predicted their graded drop in accuracy after lesioning (face-selective r=0.58; scene-selective r=0.71; body-selective r=0.52; word-selective r=0.55), suggesting that these units effectively form dissociable “content channels” that are involved in processing some categories more than others (e.g. deficits were anticorrelated when lesioning face vs. scene units: r=–0.23). Theoretically, these results demonstrate that high activations are indicative of functional role, and that the tuning directions of a distributed population code are not rotationally arbitrary. Based on this “positive routing” theory, we developed a novel sparse positive-weighted readout scheme (in contrast to standard fully connected readout), and found the same recognition deficits due to lesions held under this more efficient procedure. Broadly, this work helps reconcile existing perspectives on the functional characteristics of object-responsive cortex, proposing that information is processed via positive activation through content-specific channels along the visual hierarchy, with sparse positive readout as a potential mode of efficient information transmission to downstream areas.

Acknowledgements: This research was supported by NSF CAREER BCS-1942438.