Cross-Attentional Audio-Visual Fusion for Weakly-Supervised Action Localization

Juntae Lee · Mihir Jain · Hyoungwoo Park · Sungrack Yun

Keywords: [ Audio-Visual ] [ Multimodal Attention ] [ Action localization ] [ Event localization ] [ Weak-supervision ]

[ Abstract ]
[ Paper ]
Thu 6 May 5 p.m. PDT — 7 p.m. PDT


Temporally localizing actions in videos is one of the key components for video understanding. Learning from weakly-labeled data is seen as a potential solution towards avoiding expensive frame-level annotations. Different from other works which only depend on visual-modality, we propose to learn richer audiovisual representation for weakly-supervised action localization. First, we propose a multi-stage cross-attention mechanism to collaboratively fuse audio and visual features, which preserves the intra-modal characteristics. Second, to model both foreground and background frames, we construct an open-max classifier that treats the background class as an open-set. Third, for precise action localization, we design consistency losses to enforce temporal continuity for the action class prediction, and also help with foreground-prediction reliability. Extensive experiments on two publicly available video-datasets (AVE and ActivityNet1.2) show that the proposed method effectively fuses audio and visual modalities, and achieves the state-of-the-art results for weakly-supervised action localization.

Chat is not available.