Recent improvements in deep learning for remote sensing have shown that it is possible to detect individual trees using high resolution satellite remote sensing data. However, there has not been an evaluation of the robustness of individual tree detection methods to distribution shifts across varying geographies, and this limits the applicability of these methods to diverse areas beyond the sites in which they were trained. To address this, we introduce a benchmark dataset comprising varying agro-ecological zones for remote sensing tree detection in agroforestry farms in India. We then use this dataset to conduct a geographic robustness evaluation of out-of-distribution performance of different deep learning approaches for remote sensing tree detection. Results indicate strong performance of deep learning in detecting trees under conventional evaluation, yet a significant drop in performance in out-of-distribution agro-ecological zones for baseline methods. We report some improvements with foundation model based approaches including SAM and Grounding DINO, but find that they also exhibit similar performance drops out-of-distribution. Our study pushes the boundaries of current research by challenging machine learning methods with a dataset and evaluation protocol that better represents real-world variability, shedding light on the robustness and adaptability of different individual tree detection methods.