Skip to yearly menu bar Skip to main content


Workshop

Learning Invariance with Compact Transforms

Anna Thomas · Albert Gu · Tri Dao · Atri Rudra · Christopher Re

East Meeting Level 8 + 15 #9

Mon 30 Apr, 4:30 p.m. PDT

The problem of building machine learning models that admit efficient representations and also capture an appropriate inductive bias for the domain has recently attracted significant interest. Existing work for compressing deep learning pipelines has explored classes of structured matrices that exhibit forms of shift-invariance akin to convolutions. We leverage the displacement rank framework to automatically learn the structured class, allowing for adaptation to the invariances required for a given dataset while preserving asymptotically efficient multiplication and storage. In a setting with a small fixed parameter budget, our broad classes of structured matrices improve final accuracy by 5-7% on standard image classification datasets compared to conventional parameter constraining methods.

Live content is unavailable. Log in and register to view live content