Long Ni¹, Michael S. Landy¹; ¹New York University

When confronted with an influx of sensory inputs, our visual system often relies on ensemble coding to rapidly extract summary statistics from a spatial array of stimuli. A prevailing view in the field is that observers use only a small, selective subset of the items—roughly the square root of the ensemble size—to compute these summary estimates (Whitney & Yamanashi Leib, 2018). Proponents for this subsampling strategy argue for its “efficiency”, as integrating information from a small subset would presumably require less resource expenditure, and for its “efficacy”, as observer models that employ such a strategy can quantitatively predict human performance of ensemble coding across varying set sizes. Here, we revisit both aspects of the subsampling strategy. The prediction that integrating a few items is sufficient to capture human data is based on a problematic assumption. It assumes that sensory encoding noise of each ensemble item remains constant across set sizes. A corollary of this assumption is that the total resource expenditure—measured in Fisher information—scales linearly with set size in ensemble coding. Under a more realistic assumption that the total coding resources are limited, such that the encoding precision for each item decreases as set size increases, we demonstrate that an observer model that computes summary statistics from the entire stimulus ensemble can achieve equal decision accuracy with a consistently lower total resource budget (i.e., lower Fisher information) for any set size N > 1. We further show that this resource-constrained full-integration model can better account for data from Baek & Chong (2020) in which subjects discriminated the average item size in ensembles with varying set sizes. Our analysis challenges the “efficiency” argument for this subsampling strategy and opens the door to further evaluating its “efficacy” against the resource-constrained model in explaining existing data.

Acknowledgements: NIH EY08266, NYUAD Center for Brain and Health, funded by Tamkeen under NYU Abu Dhabi Research Institute award CG012