Martin A. Giese¹, Xinrui Jiang¹; ¹Hertie Institute / CIN, University Clinic Tuebingen, Germany

Visual perception is modulated by concurrent action execution, and action execution can be modulated by concurrent action vision. This mutual dependence can be captured by dynamically coupled neural representations for perceived actions in the visual pathway, and for planned actions in the motor pathway. The computational mechanisms that link such dynamic representations are poorly understood, and popular data-driven large-scale neural network models, due to their complexity, do not allow a deeper mathematical analysis of the relevant interactions. METHODS: We present a dynamic neural model for the coupling between action perception and action execution of hand movements. It is based on coupled nonlinear dynamic neural fields / neural mass models. Input movies are mapped onto inputs of vision fields by a feed-forward deep neural network that accomplishes hand pose recognition on sequences of real images. This level is coupled to the second dynamic level of the model, which is based on motor fields that represent hand actions in terms of self-generated propagating activation patterns. We study how such representations can be learned by time-dependent Hebbian learning, and we mathematically analyze the dynamics of the individual and the coupled representations. RESULTS: The model reproduces several experimental results on perception-action coupling within a unifying framework (influence of action execution of visual action detection, influence of concurrent action vision on accuracy of action execution, influence of baseline delay on the perception of delay changes between action perception and execution). In addition, it provides a mathematical understanding of why these interactions occur, and how they might emerge through learning. CONCLUSIONS: Neural mass models are meaningful tools to understand dynamic interactions in multi-level dynamic cortical representations in the perception-action loop. Opposed to data-driven standard neural network models they allow an understanding of the underlying dynamical mechanisms.

Acknowledgements: The work was funded by ERC 2019-SyG-RELEVANCE-856495. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Xinrui Jiang.