Johannes Burge¹; ¹University of Pennsylvania

A fundamental problem of perception is to estimate behaviorally relevant properties of the environment, and the organism’s relation to it, from highly variable and uncertain stimuli. Here, we report a new class of image-computable ideal observers—spectral ideal observers—that optimally solve such problems. Spectral ideal observers are distinguished from more traditional ideal observers because they do not assume target stimuli with a stereotyped spatial structure; rather, they process stimuli with random spatial structure as perceptual systems must in natural viewing. Further, they operate on measurements of the real and imaginary coefficients of a stimulus’ Fourier transform—rather than on amplitude and phase, or on pixels—, by characterizing the joint probability distributions of these coefficients for different states of the latent variable of interest. For filtered (e.g. 1/f) noise stimuli—which share statistical properties with natural images—the coefficients are conditionally Gaussian; for natural images and sounds, appropriate normalization Gaussianizes them. From these distributions, the posterior over the latent variable, estimates, and/or the optimal Bayesian-theoretic decision variable can be computed. The results provide principled performance predictions, and reveal the receptive fields and energy-model-like neural computations that optimally support performance. We develop a spectral ideal observer for estimating focus error and pupil size from signals in the cone responses to small patches (0.5deg) of images blurred by the optics of individual human eyes. Estimates are accurate and precise. Hence, image-based estimates of focus error (and pupil size) could be used by circuits that drive human accommodation. We discuss the design of experiments that will test novel predictions of this theory. We additionally develop spectral ideal observers for other basic tasks in visual and auditory science: 3D surface attitude (i.e. slant & tilt) estimation, binocular disparity estimation with different optics in the two eyes, and binaural sound localization (elevation & azimuth).

Acknowledgements: This work was supported by the National Eye Institute and the Office of Behavioral and Social Sciences Research, National Institutes of Health Grant R01-EY028571 to J.B.