Training neural networks for large-scale land cover classification from satellite imagery requires extensive labels for training and evaluation. While most methods are designed around dense annotations, another promising idea is to rely on sparse labels, such as openly available in-situ data. However, these data pose challenges in terms of model design and training. In this paper, we present a specially designed neural network architecture for sparsely labeled land cover classification from Sentinel-2 images and LUCAS data. Our network is a variant of Graph U-Net which represents images as graphs and uses transformer-inspired graph convolutional layers and pooling layers based on hierarchical image oversegmentations. Additionally, we adapt deep bilateral filtering modules to this architecture. In our experiments, we demonstrate that our network is able to learn from sparse labels more efficiently than traditional approaches, outperforming standard U-Nets.