When Does Second-Order Optimization Speed Up Training?
Satoki Ishikawa · Rio Yokota
2024 Poster
in
Affinity Event: Tiny Papers Poster Session 4
in
Affinity Event: Tiny Papers Poster Session 4
Abstract
While numerous second-order optimization methods have been proposed to accelerate training in deep learning, they are seldom used in practice. This is partly due to a limited understanding of the conditions under which second-order optimization outperforms first-order optimization. This study aims to identify these conditions, particularly in terms of batch size and dataset size.We find empirically that second-order optimization outperforms first-order optimization when the batch size is large and the data set size is not too large.
Video
Chat is not available.
Successful Page Load