Poster
Few-Shot Learning via Learning the Representation, Provably
Simon Du · Wei Hu · Sham M Kakade · Jason Lee · Qi Lei
Virtual
Keywords: [ representation learning ] [ statistical learning theory ]
Abstract:
This paper studies few-shot learning via representation learning, where one uses TT source tasks with n1n1 data per task to learn a representation in order to reduce the sample complexity of a target task for which there is only n2(≪n1)n2(≪n1) data. Specifically, we focus on the setting where there exists a good common representation between source and target, and our goal is to understand how much a sample size reduction is possible. First, we study the setting where this common representation is low-dimensional and provide a risk bound of ˜O(dkn1T+kn2)~O(dkn1T+kn2) on the target task for the linear representation class; here dd is the ambient input dimension and k(≪d)k(≪d) is the dimension of the representation. This result bypasses the Ω(1T)Ω(1T) barrier under the i.i.d. task assumption, and can capture the desired property that all n1Tn1T samples from source tasks can be \emph{pooled} together for representation learning. We further extend this result to handle a general representation function class and obtain a similar result. Next, we consider the setting where the common representation may be high-dimensional but is capacity-constrained (say in norm); here, we again demonstrate the advantage of representation learning in both high-dimensional linear regression and neural networks, and show that representation learning can fully utilize all n1Tn1T samples from source tasks.
Chat is not available.