Beyond prediction: Interpretable brain encoding models for understanding vision

Symposium: Friday, May 15, 2026, 1:45 – 3:45 pm, Talk Room 1

Schedule of Events | Search Abstracts | Symposia | Talk Sessions | Poster Sessions

Organizers: Zitong Lu¹, Apurva Ratan Murty²; ¹Massachusetts Institute of Technology, ²Georgia Institute of Technology
Presenters: Hossein Adeli, Katharina Dobs, Iris I.A. Groen, Zitong Lu, Andrew Luo, Apurva Ratan Murty

A central challenge in vision science is to understand how visual inputs are transformed into neural representations – and critically, how computational models can reveal the mechanisms underlying this transformation. Although neural encoding models have advanced rapidly with deep learning and large-scale datasets, purely performance-driven approaches offer limited scientific insight. To genuinely advance vision science, encoding models must be interpretable - constructed in ways that make their internal representations, architectural assumptions, and computational operations transparent and meaningful for understanding the brain. This symposium brings together early-career researchers using interpretable brain encoding models as mechanistic tools, highlighting both how such models are constructed and how they can be used – beyond traditional cognitive neuroscience methods – to reveal the mechanisms of visual processing. Across fMRI, EEG, and cross-subject datasets, the speakers will demonstrate how carefully designed architectures, structured feature spaces, representational analyses, and in-silico manipulations can uncover aspects of visual computation that conventional approaches cannot isolate. Together, the talks illustrate converging directions within this emerging landscape. Some approaches embed biologically meaningful structure, such as retinotopic maps or category-selective pathways, enabling models to recapitulate known functional organization while generating new hypotheses about information flow. Others interrogate internal representations through model–brain alignment to infer how visual categories and fine-grained feature dimensions may be encoded in the human visual system. Several studies leverage generative or perturbation-based encoding frameworks, which allow direct manipulation of internal representations to identify candidate computations underlying specific neural responses. Still others explore cross-subject generalization, using interpretable alignment strategies to reveal shared representational structure despite individual variability. The six presentations span these perspectives in a coherent progression. Hossein Adeli will begin by introducing an interpretable attention-based routing model that links retinotopic features to category-selective regions. Also focusing on category-selectivity, Katharina Dobs will show how large-scale comparisons between ANNs and human cortex reveal the organizing principles of functional specialization. Extending beyond object vision, Iris Groen will demonstrate how encoding models can uncover scene affordance and dynamic video perception through model-brain alignment. Shifting from fMRI to EEG, Zitong Lu will present Img2EEG, a generative encoding framework enabling mechanistic exploration of temporal dynamics through simulated EEG signals. Then, Andrew Luo will illustrate how meta-learned cross-subject encoding and decoding models reveal generalizable representational structure across individuals. Finally, Apurva Ratan Murty will show the strengths and limits of current ANN-based encoding models by conducting cognitive experiments in them. All these talks present a unified vision: interpretable encoding models are becoming scientific instruments – tools that help reveal the computations that give rise to it. By integrating architectural constraints, representational transparency, and model-based experimentation, these approaches open new pathways for probing visual mechanisms in naturalistic settings. This symposium, featuring a brief introduction followed by six 16–20 minute talks with discussion, is designed to engage a broad audience across experimental and computational domains. By highlighting how interpretable brain encoding models can overcome limitations of traditional methods and provide mechanistic insight, the session aims to inspire new ways of using models to understand how the brain constructs visual experience.

Talk 1

Dynamic transformer routing from retinotopic cortex explains encoding in category-selective cortical areas

Hossein Adeli¹, Nikolaus Kriegeskorte¹; ¹Columbia University

A major goal of neuroscience is to understand brain computations during naturalistic visual perception. A dominant approach is to use image-computable deep neural networks as a basis for linear encoding models. However, this approach ignores the structure of the feature maps both in the brain and the models, which could be used to build more interpretable models. In this work, we employ the transformer attention mechanism to model how retinotopic visual features can be dynamically routed to category-selective areas in high-level visual processing. We show that this computational motif not only is significantly more powerful than alternative methods in predicting brain activity during natural scene viewing, it is also inherently more interpretable because the attention routing signals can be easily visualized for any input image. Analyzing the attention signals in our model replicates the categorial selectivity of known regions (e.g. image patches with faces are routed to predict fMRI voxels in the Fusiform Face Area, FFA) for face, body, word, and place regions and reveals structured between-area visual and semantic similarities that reflect their underlying computations. We also show how this state-of-the-art encoding model can serve as a digital twin, on which we perform in silico experiments to discover categorical selectivity across the whole brian. Our results suggest that the human visual hierarchy employs a dynamic routing mechanism that guides relevant information from image-specific retinal locations to category-selective regions. Our work shows how interpretable encoding models can help us understand visual processing in the brain.

Talk 2

Interpreting vision through the lens of category selectivity in brains and ANNs

Katharina Dobs¹; ¹Justus Liebig University Giessen

Functional specialization is a hallmark of visual processing. In the human visual cortex, distinct regions respond selectively to specific categories, such as faces, bodies, or scenes. Strikingly, artificial neural networks (ANNs) for vision show emergent patterns of category selectivity reminiscent of those observed in the brain. While this functional alignment offers a unique opportunity to probe the origins of category selectivity, many questions remain open: To what extent do ANNs and the human visual cortex rely on shared organizing principles, and can ANNs help reveal functional specializations that have not yet been discovered in the brain? Here, we systematically compare category selectivity in ANNs and human visual cortex across a large and diverse set of categories. We find that the degree of selectivity for a given category in ANNs predicts its selectivity in the visual cortex, particularly for ANN units and cortical regions showing clear functional tuning. By quantifying the correspondence between category selectivity in artificial and biological vision systems, these results offer an interpretable framework to investigate the computational principles underlying functional specialization and to uncover previously unknown selectivities in the visual cortex.

Talk 3

Leveraging DNN-brain alignment to understand how humans perceive scenes and videos

Iris I.A. Groen¹; ¹University of Amsterdam

The unprecedented ability of deep neural networks (DNNs) to predict neural responses in the human visual cortex has created excitement in vision research about these models’ potential to capture human visual perception. However, many initial observations relating the inner representations of DNNs to those in the human brain primarily confirmed existing ideas about visual cortex (e.g., the presence of a low-to-high complexity object feature hierarchy). Can we leverage brain-DNN alignment to learn something new about how human vision works? In this talk, I will show how we use DNN-based representational alignment and model comparison to move from object vision towards new domains, including the neural representation of scene affordances and spatial and temporal neural dynamics during video perception. Moreover, I will showcase how DNN-brain alignment can be leveraged to generate new hypotheses about neural representations in human visual cortex via brain-guided image diffusion, as implemented in our newly developed Brain Activation Control Through Image Variation (BrainACTIV) method.

Talk 4

Exploring human vision through Img2EEG: An encoding framework generating high-resolution temporal EEG signals from visual inputs

Zitong Lu^1,2; ¹The Ohio State University, ²Massachusetts Institute of Technology

Understanding the complex interplay between visual stimuli and brain activity has been a focal point in cognitive neuroscience. The recent advent of image-to-brain encoding models presents an unparalleled opportunity for deeper explorations and broader applications in experimental and computational research. Building on this momentum, we developed “Img2EEG”, a pioneering encoding framework consisting of visual, semantic and integration modules based on state-of-the-art deep learning models from computer vision and natural language processing. Trained on a large-scale EEG dataset of natural images at the individual subject level, Img2EEG effectively generates highly realistic EEG signals given any image input. We propose Img2EEG as an innovative and internally manipulable tool for investigating visual mechanisms. Through internal and external ablation experiments on Img2EEG, we unfold the detailed temporal dynamics underlying visual processes. Moreover, through in silico experiments on Img2EEG, by feeding it novel image sets outside its training data, we not only reproduced the classic N170 component in the model-generated EEG signals but also leveraged its internal manipulability to identify the key features driving this response. Furthermore, our Img2EEG achieves more accurate decoding, outperforming current state-of-the-art EEG decoding models (e.g., 200-class classification accuracy on THINGS EEG2). Finally, we applied Img2EEGs across all ImageNet images, creating a large stimulated EEG dataset, ‘ImageNet-SimEEG’, providing potential applications for decoding images from EEG and advancing more brain-like visual models. Overall, Img2EEG mapping from visual inputs to high temporal resolution brain signals offers novel and powerful approaches to probe human visual representations.

Talk 5

Meta-learning in context enables training-free cross-subject encoders and decoders

Andrew Luo¹; ¹University of Hong Kong

Understanding the relationship between sensory stimuli and neural responses is a fundamental goal in neuroscience. A significant obstacle has been developing models that generalize across individuals due to high inter-subject variability in neural representations, traditionally necessitating bespoke, subject-specific models. To address this, we propose a unified framework that leverages in-context learning for both predicting neural responses (encoding) and interpreting them (decoding), without requiring subject-specific fine-tuning. Our approach utilizes a transformer-based architecture, meta-optimized to rapidly infer an individual's unique neural encoding patterns by conditioning on a small set of stimulus-response examples. For encoding, this allows the model to accurately predict voxelwise neural responses to entirely novel stimuli, outperforming existing methods in low-data regimes. For decoding, the framework performs hierarchical functional inversion to achieve robust semantic interpretation of brain activity. The model demonstrates strong generalization across different subjects and acquisition parameters, a critical advance that eliminates the need for anatomical alignment or stimulus overlap. This work marks a significant step away from single-subject analysis towards a generalizable foundation model for non-invasive brain signal analysis. By enhancing interpretability and predictive power, our framework advances the broader effort to predict and understand neural representations in the human brain.

Talk 6

Using cognitive tests to reveal the strengths and limits of brain encoding models

Apurva Ratan Murty¹; ¹Georgia Institute of Technology

If artificial neural networks (ANNs) are to serve as mechanistic hypotheses of the visual system, they must also recapitulate decades of cognitive experiments. Neural predictivity, the current modeling standard, is useful but does not tell us the computations a model implements. Cognitive neuroscience studies, by contrast, were designed precisely as probes of perceptual computations, yet we almost never use them to evaluate brain models. In this work, we revisited 20 experiments across face, scene, and body perception that localized and characterized responses in human category-selective cortex. Many of these studies used manipulated stimuli that fall outside the training domain of standard ANNs. We ran these experiments directly through models (N = 150) without retraining them on any of the specific images or fMRI data from previous studies. As expected, ANNs recapitulated many hallmark univariate and multivariate signatures. However, the cognitive tests also exposed clear differences between brain regions, between specific cognitive phenomena, and separated the 150 models better than prediction scores alone. Together, these findings show how cognitive neuroscience experiments can clarify what current ANN models explain and reveal their limits as in-silico models of the brain. All models are released for researchers to run their own cognitive tests.