Skip to yearly menu bar Skip to main content


In-Person Poster presentation / poster accept

Learning Locality and Isotropy in Dialogue Modeling

Han Wu · Haochen Tan · Mingjie Zhan · Gangming Zhao · Shaoqing Lu · Ding Liang · Linqi Song

MH1-2-3-4 #31

Keywords: [ representation learning ] [ feature space calibration ] [ dialogue system ] [ Applications ]


Abstract:

Existing dialogue modeling methods have achieved promising performance on various dialogue tasks with the aid of Transformer and the large-scale pre-trained language models. However, some recent studies revealed that the context representations produced by these methods suffer the problem of anisotropy. In this paper, we find that the generated representations are also not conversational, losing the conversation structure information during the context modeling stage. To this end, we identify two properties in dialogue modeling, i.e., locality and isotropy, and present a simple method for dialogue representation calibration, namely SimDRC, to build isotropic and conversational feature spaces. Experimental results show that our approach significantly outperforms current state-of-the-art models on three open-domain dialogue tasks with eight benchmarks. More in-depth analyses further confirm the effectiveness of our proposed approach. We release the code at https://github.com/hahahawu/SimDRC.

Chat is not available.