Skip to yearly menu bar Skip to main content


A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis

DIPANJYOTI PAUL · Arpita Chowdhury · Xinqi Xiong · Feng-Ju Chang · David Carlyn · Samuel Stevens · Kaiya Provost · Anuj Karpatne · Bryan Carstens · Daniel Rubenstein · Charles Stewart · Tanya Berger-Wolf · Yu Su · Wei-Lun Chao

Halle B #245
[ ] [ Project Page ]
Tue 7 May 1:45 a.m. PDT — 3:45 a.m. PDT


We present a novel usage of Transformers to make image classification interpretable. Unlike mainstream classifiers that wait until the last fully connected layer to incorporate class information to make predictions, we investigate a proactive approach, asking each class to search for itself in an image. We realize this idea via a Transformer encoder-decoder inspired by DEtection TRansformer (DETR). We learn ''class-specific'' queries (one for each class) as input to the decoder, enabling each class to localize its patterns in an image via cross-attention. We name our approach INterpretable TRansformer (INTR), which is fairly easy to implement and exhibits several compelling properties. We show that INTR intrinsically encourages each class to attend distinctively; the cross-attention weights thus provide a faithful interpretation of the prediction. Interestingly, via ''multi-head'' cross-attention, INTR could identify different ''attributes'' of a class, making it particularly suitable for fine-grained classification and analysis, which we demonstrate on eight datasets. Our code and pre-trained models are publicly accessible at the Imageomics Institute GitHub site:

Chat is not available.