In-Person Oral presentation / top 25% paper

AANG : Automating Auxiliary Learning

Lucio Dery · Paul Michel · Mikhail Khodak · Graham Neubig · Ameet Talwalkar

AD11
[ Abstract ] [ Livestream: Visit Oral 4 Track 1: Unsupervised and Self-supervised learning ]
Tue 2 May 6:10 a.m. — 6:20 a.m. PDT

Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning. Whilst much work has been done to formulate useful auxiliary objectives, their construction is still an art which proceeds by slow and tedious hand-design. Intuition for how and when these objectives improve end-task performance has also had limited theoretical backing. In this work, we present an approach for automatically generating a suite of auxiliary objectives. We achieve this by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. Next, we theoretically formalize widely-held intuitions about how auxiliary learning improves generalization on the end-task. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task.With natural language processing (NLP) as our domain of study, we demonstrate that our automated auxiliary learning pipeline leads to strong improvements over competitive baselines across continued training experiments on a pre-trained model on 5 NLP end-tasks.

Chat is not available.