Zitong Lu¹, Shuning Tang², Dongwei Li²; ¹Massachusetts Institute of Technology, ²Beijing Normal University

Understanding how the human brain processes dynamic audiovisual events requires datasets that combine naturalistic stimulation, high temporal resolution, and systematic experimental manipulations. Here we introduce Moments-EEG, the first large-scale, high temporal resolution human EEG dataset of naturalistic video events. Across participants, we collected 64-channel scalp EEG during free-viewing of short (3-second) naturalistic video clips drawn from the Audiovisual Moments in Time dataset. The stimulus set comprised 896 training videos (16 event types × 56 video per type) presented three times, and to a test set of 64 videos (16 types × 4 videos per type) repeated 40 times under three conditions: (1) intact audiovisual videos; (2) intact visuals paired with scrambled meaningless audio, and (3) scrambled meaningless videos with intact audio. Each training clip was repeated three times and each test clip was repeated 40 times, yielding 10,368 trials per participant. Moments-EEG complements existing short-video fMRI datasets by capturing neural dynamics of audiovisual event perception at millisecond precision across the whole brain. Moments-EEG enables several key scientific advances. First, this dataset serves as a benchmark for video-to-EEG encoding and EEG-to-video decoding models of complex events, supporting rigorous evaluation of how computational features align with neural dynamics. Second, the high trial counts for naturalistic audiovisual events allow precise characterization of the temporal evolution of information processing in dynamic scenes. Third, the audiovisual manipulations isolate modality-specific processing. Comparing intact versus scrambled conditions disentangles the individually and jointly contributions of auditory and visual inputs. Forth, Moments-EEG provides a testbed for evaluating the temporal alignment between artificial neural networks and the human brain under stimulation with rich annotations. Overall, by combining naturalistic stimuli, multisensory perturbations, repeated-measurements, and behavioral annotations, Moments-EEG offers a rich resource for study human event perception, multisensory integration, and advancing temporally resolved brain-model alignments.

Acknowledgements: National Natural Science Foundation of China (32400863)