Skip to yearly menu bar Skip to main content

In-Person Poster presentation / top 25% paper

Simplicial Embeddings in Self-Supervised Learning and Downstream Classification

Samuel Lavoie · Christos Tsirigotis · Max Schwarzer · Ankit Vani · Mikhail Noukhovitch · Kenji Kawaguchi · Aaron Courville

MH1-2-3-4 #145

Keywords: [ Unsupervised and Self-supervised learning ] [ pre-training ] [ self-supervised learning ] [ representation learning ]

Abstract: Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into $L$ simplices of $V$ dimensions each using a \texttt{softmax} operation. This procedure conditions the representation onto a constrained space during pretraining and imparts an inductive bias for group sparsity. For downstream classification, we formally prove that the SEM representation leads to better generalization than an unnormalized representation.Furthermore, we empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR-100 and ImageNet. Finally, when used in a downstream classification task, we show that SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically-relevant classes.

Chat is not available.