Skip to yearly menu bar Skip to main content


ContextRef: Evaluating Referenceless Metrics for Image Description Generation

Elisa Kreiss · Elisa Kreiss · Eric Zelikman · Christopher Potts · Nick Haber

Halle B #56
[ ]
Wed 8 May 1:45 a.m. PDT — 3:45 a.m. PDT


Referenceless metrics (e.g., CLIPScore) use pretrained vision--language models to assess image descriptions directly without costly ground-truth reference texts. Such methods can facilitate rapid progress, but only if they truly align with human preference judgments. In this paper, we introduce ContextRef, a benchmark for assessing referenceless metrics for such alignment. ContextRef has two components: human ratings along a variety of established quality dimensions, and ten diverse robustness checks designed to uncover fundamental weaknesses. A crucial aspect of ContextRef is that images and descriptions are presented in context, reflecting prior work showing that context is important for description quality. Using ContextRef, we assess a variety of pretrained models, scoring functions, and techniques for incorporating context. None of the methods is successful with ContextRef, but we show that careful fine-tuning yields substantial improvements. ContextRef remains a challenging benchmark though, in large part due to the challenge of context dependence.

Chat is not available.