In confronting the pressing issue of climate change, we introduce "CausalPrompt", an innovative prompting strategy that adapts large language models (LLMs) for classification and regression tasks through the application of weakly supervised causal reasoning. We delve into the complexities of data shifts within energy systems, often resulting from the dynamic evolution of sensor networks, leading to discrepancies between training and test data distributions or feature inconsistencies. By embedding domain-specific reasoning in the finetuning process, CausalPrompt significantly bolsters the adaptability and resilience of energy systems to these shifts. We show that CausalPrompt significantly enhances predictions in scenarios characterized by feature shifts, including electricity demand, solar power generation, and cybersecurity within energy infrastructures. This approach underlines the crucial role of CausalPrompt in enhancing the reliability and precision of predictions in energy systems amid feature shifts, highlighting its significance and potential for real-world applications in energy management and cybersecurity, contributing effectively to climate change mitigation efforts.