Skip to yearly menu bar Skip to main content

In-Person Poster presentation / top 25% paper

AANG : Automating Auxiliary Learning

Lucio Dery · Paul Michel · Mikhail Khodak · Graham Neubig · Ameet Talwalkar

MH1-2-3-4 #162

Keywords: [ Algorithmic Stability ] [ meta-learning ] [ automl ] [ natural language processing ] [ Auxiliary Learning ] [ multitask learning ] [ Unsupervised and Self-supervised learning ]


Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning. Whilst much work has been done to formulate useful auxiliary objectives, their construction is still an art which proceeds by slow and tedious hand-design. Intuition for how and when these objectives improve end-task performance has also had limited theoretical backing. In this work, we present an approach for automatically generating a suite of auxiliary objectives. We achieve this by deconstructing existing objectives within a novel unified taxonomy, identifying connections between them, and generating new ones based on the uncovered structure. Next, we theoretically formalize widely-held intuitions about how auxiliary learning improves generalization on the end-task. This leads us to a principled and efficient algorithm for searching the space of generated objectives to find those most useful to a specified end-task.With natural language processing (NLP) as our domain of study, we demonstrate that our automated auxiliary learning pipeline leads to strong improvements over competitive baselines across continued training experiments on a pre-trained model on 5 NLP end-tasks.

Chat is not available.