Poster
Smooth Loss Functions for Deep Top-k Classification
Leonard Berrada · Andrew Zisserman · M. Pawan Kumar
East Meeting level; 1,2,3 #26
[
Abstract
]
Abstract:
The top- error is a common measure of performance in machine learning and computer vision. In practice, top- classification is typically performed with deep neural networks trained with the cross-entropy loss. Theoretical results indeed suggest that cross-entropy is an optimal learning objective for such a task in the limit of infinite data. In the context of limited and noisy data however, the use of a loss function that is specifically designed for top- classification can bring significant improvements.
Our empirical evidence suggests that the loss function must be smooth and have non-sparse gradients in order to work well with deep neural networks. Consequently, we introduce a family of smoothed loss functions that are suited to top- optimization via deep learning. The widely used cross-entropy is a special case of our family. Evaluating our smooth loss functions is computationally challenging: a na{\"i}ve algorithm would require operations, where is the number of classes. Thanks to a connection to polynomial algebra and a divide-and-conquer approach, we provide an algorithm with a time complexity of . Furthermore, we present a novel approximation to obtain fast and stable algorithms on GPUs with single floating point precision. We compare the performance of the cross-entropy loss and our margin-based losses in various regimes of noise and data size, for the predominant use case of . Our investigation reveals that our loss is more robust to noise and overfitting than cross-entropy.
Live content is unavailable. Log in and register to view live content