Poster
Learning One-hidden-layer Neural Networks with Landscape Design
Rong Ge · Jason Lee · Tengyu Ma
East Meeting level; 1,2,3 #13
[
Abstract
]
Abstract:
We consider the problem of learning a one-hidden-layer neural network: we assume the input x is from Gaussian distribution and the label , where a is a nonnegative vector and is a full-rank weight matrix, and is a noise vector. We first give an analytic formula for the population risk of the standard squared loss and demonstrate that it implicitly attempts to decompose a sequence of low-rank tensors simultaneously.
Inspired by the formula, we design a non-convex objective function whose landscape is guaranteed to have the following properties:
1. All local minima of are also global minima.
2. All global minima of correspond to the ground truth parameters.
3. The value and gradient of can be estimated using samples.
With these properties, stochastic gradient descent on provably converges to the global minimum and learn the ground-truth parameters. We also prove finite sample complexity results and validate the results by simulations.
Live content is unavailable. Log in and register to view live content