In machine learning for geographic data, we often observe differences in data availability and distribution shifts across distinct geographic units, e.g., continents. This is a common challenge in remote sensing tasks, such as crop yield forecasting or flood mapping. In many of these scenarios, we have models trained on a data-rich region and apply domain adaptation to transfer predictive capabilities to the target region. However, the effectiveness of domain transfer can suffer from distribution shifts, posing critical challenges for model deployment. In this work, we show that, even in the absence of labels, certain domain distance measures, based on image and location embeddings, can serve as a proxy measure for transfer performance. We highlight this capacity on a set of real-world geographic adaptation datasets. We also outline further avenues for leveraging these domain distances to improve model generalization, through principled dataset design.