Processing math: 100%
Skip to yearly menu bar Skip to main content


Spotlight Poster

On the Provable Advantage of Unsupervised Pretraining

Jiawei Ge · Shange Tang · Jianqing Fan · Chi Jin

Halle B #192

Abstract: Unsupervised pretraining, which learns a useful representation using a large amount of unlabeled data to facilitate the learning of downstream tasks, is a critical component of modern large-scale machine learning systems. Despite its tremendous empirical success, the rigorous theoretical understanding of why unsupervised pretraining generally helps remains rather limited---most existing results are restricted to particular methods or approaches for unsupervised pretraining with specialized structural assumptions. This paper studies a generic framework,where the unsupervised representation learning task is specified by an abstract class of latent variable models Φ and the downstream task is specified by a class of prediction functions Ψ. We consider a natural approach of using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for learning downstream tasks. We prove that, under a mild informative'' condition, our algorithm achieves an excess risk of tildemathcalO(C_Φ/m+C_Ψ/n) for downstream tasks, where C_Φ,C_Ψ are complexity measures of function classes Φ,Ψ, and m,n are the number of unlabeled and labeled data respectively. Comparing to the baseline of ˜O(C_ΦΨ/n) achieved by performing supervised learning using only the labeled data, our result rigorously shows the benefit of unsupervised pretraining when mn and C_ΦΨ>C_Ψ. This paper further shows that our generic framework covers a wide range of approaches for unsupervised pretraining, including factor models, Gaussian mixture models, and contrastive learning.

Chat is not available.