Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention

• Published in: IEEE transactions on multimedia Vol. 15; no. 7; pp. 1553 - 1568
• Main Authors: Evangelopoulos, Georgios, Zlatintsi, Athanasia, Potamianos, Alexandros, Maragos, Petros, Rapantzikos, Konstantinos, Skoumas, Georgios, Avrithis, Yannis
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.11.2013; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Applied sciences; Artificial intelligence; Attention; audio saliency; Biological and medical sciences; Computational modeling; Computer science; control theory; systems; Cues; Data processing. List processing. Character string processing; Exact sciences and technology; Feature extraction; Fundamental and applied biological sciences. Psychology; fusion; Memory organisation. Data processing; Modulation; Motion pictures; movie summarization; multimodal saliency; multistream processing; Pattern recognition. Digital image processing. Computational geometry; Perception; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; Semantics; Software; Speech and sound recognition and synthesis. Linguistics; Streaming media; Streams; Task analysis; text saliency; video summarization; Vision; Visual; visual saliency; Visualization; Waveforms; Space time correlation; Tracking; multimodal saliency; Video signal; Modeling; Audiovisual; multistream processing; Linguistics; Selection criterion; text saliency; Multimodal interface; Audiovisual equipment; audio saliency; Pattern extraction; video summarization; Textual data; Streaming; Computer vision; Abstract; movie summarization; Attention; Model driven architecture; Text; Annotation; Grammatical inference; fusion; Dimension reduction; Cognitive theory; Hearing; visual saliency; Multimodality; Bottom up method; Stimulus salience; Data fusion; Feature extraction; Visual information; Visual attention
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2013.2267205

Multimodal streams of sensory information are naturally parsed and integrated by humans using signal-level feature extraction and higher level cognitive processes. Detection of attention-invoking audiovisual segments is formulated in this work on the basis of saliency models for the audio, visual, and textual information conveyed in a video stream. Aural or auditory saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color, and orientation. Textual or linguistic saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The individual saliency streams, obtained from modality-depended cues, are integrated in a multimodal saliency curve, modeling the time-varying perceptual importance of the composite video stream and signifying prevailing sensory events. The multimodal saliency representation forms the basis of a generic, bottom-up video summarization algorithm. Different fusion schemes are evaluated on a movie database of multimodal saliency annotations with comparative results provided across modalities. The produced summaries, based on low-level features and content-independent fusion and selection, are of subjectively high aesthetic and informative quality.