Airborne hyperspectral imaging has great potential for land cover mapping with very detailed nomenclatures thanks to its spectral dimension, which is highly informative about the chemical composition of matter. In the past years, the scarcity of training data in regards to the significant spectral intra-class variability has motivated the development of inductive biases for the generalization of deep learning classification models. In contrast, we investigate in this paper an orthogonal line of research which consists in integrating deductive biases derived from a priori physical knowledge into weakly supervised learning in order to improve the robustness of classification models to changes in local irradiance conditions. Our experiments on simulated and real data demonstrate the benefits of our method in terms of classification accuracy.