Anne Peiris¹, Jaykishan Patel¹, Richard F. Murray¹; ¹York University

Deep neural networks (DNNs) have revolutionized computer vision and computational modelling, and they achieve human-like accuracy in object classification tasks. However, because DNNs learn through training, little is understood about how they work. Spatial frequency is a key feature of visual stimuli, but which frequencies are critical for DNNs’ performance? Previous work found that DNNs have a critical spatial frequency band 2–4 times wider than humans in an object classification task (Subramanian et al., 2023). This may be a shortcoming of DNNs, and they might perform better if they had narrower, human-like channels. Another possibility is that DNNs’ channels are wider because they exploit more of the useful spatial frequencies in images. We used ideal observer analysis to test these hypotheses. We ran five ideal observers and 18 DNNs in a classification task using ImageNet’s validation set (1000 categories, 50 samples each). We used a critical band masking paradigm to determine the most important spatial frequencies for object classification. We chose one representative DNN from each category of architectures available in PyTorch: one ResNet, one VGG, etc. Images were presented in filtered Gaussian noise, and object classification thresholds were measured at each noise frequency to identify critical bands. DNNs’ critical bands were, on average, around two octaves wide, consistent with previous work. Interestingly, the ideal observers’ critical bands were low-pass instead of band-pass, with a 1/f fall-off. Thus, DNNs do not exploit the entire range of informative spatial frequencies in natural images. Indeed, in a separate analysis, we found that object classification thresholds in white Gaussian noise were orders of magnitude lower for ideal observers than for DNNs, meaning that DNNs’ absolute efficiency is extremely low (η ≈ 7 × 10-5). These results demonstrate that ideal observer analysis can be highly informative for understanding fundamental properties of DNNs.

Acknowledgements: Funded by an Ontario Graduate Scholarship to AP and an NSERC Discovery Grant to RFM.