Binxu Wang¹, Jacob Prince¹, Thomas Fel¹, Akshay Jagadeesh^1,2, George Alvarez¹, Cengiz Pehlevan¹, Talia Konkle¹; ¹Harvard University, ²OpenAI

Deep neural network models can achieve high accuracy in predicting visual neuronal responses to natural images. Yet, when these same models are used to generate images to control neuronal activity, their success varies dramatically. Recent systematic evaluation work (Prince, Wang, Fel, Jagadeesh, et al.) revealed three surprising findings: (1) strong predictive performance does not ensure controllability; (2) models that fail at control tend to produce high-spatial-frequency modulations that are less effective for neurons; and (3) some models that cannot drive neurons themselves can still predict which stimuli generated by other models will be effective. We introduce a theoretical explanation based on a simplified linear regression framework. Prediction depends on the inner product $\mathbf{w}^\top \mathbf{x}$ between regression weights and natural image features, whereas neural control requires moving along the weight direction $\mathbf{w}$ itself. Because natural images exhibit highly anisotropic feature covariance—with high-frequency components contributing low variance—regression tend to amplify these poorly constrained directions. This allows large changes in $\mathbf{w}$ with minimal impact on prediction, dissociating encoding accuracy from controllability. When ground-truth neural tuning lies primarily within the natural image manifold, learned weights can acquire substantial off-manifold components. Image synthesis then pushes stimuli toward unnatural directions that are ineffective in real neurons, although these same models can still evaluate the success of stimuli produced by better-aligned models, which are more on manifold. Finally, applying random matrix theory, we predict how sample size, feature dimensionality, and neuronal noise level determine which weight modes are most error-prone, and thus when control will fail. Together, this framework explains why good encoders can be poor controllers, and motivates new approaches for on-manifold feature accentuation and robust closed-loop neural control.

Acknowledgements: This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. B.W. is funded by the Kempner Research Fellowship