Hossein Adeli¹, Ethan Hwang¹, Wenxuan Guo¹, Andrew Luo², Nikolaus Kriegeskorte¹; ¹Columbia University, ²University of Hong Kong

A fine-grained account of categorical selectivity in the cortex is essential for understanding visual processing in the brain. Classic experimental studies have identified multiple category-selective regions; however, these approaches rely on preconceived hypotheses about categories. Subsequent data-driven methods have sought to address this limitation by building encoding models that predict neural activity from different image features, but they are often limited by simple, typically linear, mapping models. We propose an in silico approach for data-driven discovery of novel category-selectivity hypotheses. The brain is first divided into 1000 functional regions. We then train an encoding model that incorporates a brain-region to image-feature cross-attention mechanism, enabling efficient mappings between deep network features and semantic patterns in the brain activity. The model significantly outperforms the linear mapping across the cortex in predicting the fMRI responses in the Natural Scene Dataset. Using this encoder, we use diffusion-based image generative models and large-scale datasets (e.g. ImageNet) to synthesize and select images that maximally activate each region. These images are then used to form a label (by averaging their CLIP representations) to characterize the selectivity. We validate that our method assigns correct labels for each region by testing that label-image-similarity predicts how well the images activate the region in held-out sets and that those images match selectivity for known areas (e.g. face, place, and word areas). We then apply the method to all the other regions in the brain that were visually responsive but had no known categorical selectivity. Our approach reveals regions with complex, compositional selectivity involving diverse semantic concepts (e.g. tool use), which we validate both within and across subjects. Our work shows that using a brain encoder as a “digital twin” offers a powerful, data-driven framework for generating hypotheses about selectivity in the human brain, which can be tested in future experiments.

Acknowledgements: Research reported in this publication was supported in part by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under award numbers 1RF1NS128897 and 4R01NS128897. The content is solely the responsibility of the authors.