The Efficiency Misnomer

Mostafa Dehghani · Yi Tay · Anurag Arnab · Lucas Beyer · Ashish Vaswani

[ Abstract ]
[ Visit Poster at Spot I3 in Virtual World ] [ OpenReview
Wed 27 Apr 10:30 a.m. PDT — 12:30 p.m. PDT


Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts.Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only a few of them.In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other.We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.

Chat is not available.