Bhavin Choksi^1,2 (), David Trung Nguyen¹, Sari Saba-Sadiya^1,2, Gemma Roig^1,2; ¹Goethe University Frankfurt, Department of Computer Science, ²The Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany

Deep neural networks are increasingly used as computational models of the human visual system, yet the extent to which brain-model alignment reflects meaningful generalization beyond natural images remains unclear. While the generalization of the alignment has been investigated on the model side, how it holds when the human brain data, instead of the model, is acquired on out-of-distribution (OOD) settings remains relatively unexplored. In this work, we systematically investigate whether alignment between model representations and human visual cortex responses predicts robustness to OOD stimuli. We quantify robustness using Effective Robustness, which isolates a model’s OOD performance beyond what is expected from its in-distribution accuracy. Across 70 ImageNet-trained models, we relate Effective Robustness on multiple OOD benchmarks to representational similarity with fMRI recordings from early visual (V1, V2) and ventral stream (V4, IT) brain regions, measured using both natural and synthetic stimuli. We find that robustness under texture-based distribution shifts consistently correlates with brain alignment in early visual areas, regardless of stimulus type. In contrast, brain-model similarity in higher ventral stream areas correlates with robustness only when derived from fMRI responses to synthetic, rather than natural, stimuli. Together, these results demonstrate that the link between model-brain alignment and model generalization is currently highly dependent on both stimulus characteristics and the cortical regions considered. Our findings highlight the gaps in the current state of the brain-model alignment and the importance of considering the stimulus used when collecting neural data and using ANNs as cognitive models.

Acknowledgements: This work was funded by the Deutsche Forschungsgemeinschaft: DFG project 5368 and DFG project 539642788, RO 6458/5-1.