Skip to yearly menu bar Skip to main content

In-Person Poster presentation / poster accept

Don’t forget the nullspace! Nullspace occupancy as a mechanism for out of distribution failure

Daksh Idnani · Vivek Madan · Naman Goyal · David J Schwab · Shanmukha Ramakrishna Vedantam

MH1-2-3-4 #47

Keywords: [ Deep Learning and representational learning ]


Out of distribution (OoD) generalization has received considerable interest in recent years. In this work, we identify a particular failure mode of OoD generalization for discriminative classifiers that is based on test data (from a new domain) lying in the nullspace of features learnt from source data. We demonstrate the existence of this failure mode across multiple networks trained across RotatedMNIST, PACS, TerraIncognita, DomainNet and ImageNet-R datasets. We then study different choices for characterizing the feature space and show that projecting intermediate representations onto the span of directions that obtain maximum training accuracy provides consistent improvements in OoD performance. Finally, we show that such nullspace behavior also provides an insight into neural networks trained on poisoned data. We hope our work galvanizes interest in the relationship between the nullspace occupancy failure mode and generalization.

Chat is not available.