Granular and repeated measurements of material wealth are essential for understanding and improving economic livelihoods, but detailed survey data is scarce, especially in Africa --- the continent most affected by poverty. Recent machine learning approaches successfully leverage non-traditional data sources such as satellite data to predict material wealth across locations but not its variation over time. This work systematically investigates the potential of publicly available satellite and OpenStreetMap data to predict levels and fluctuations in annual consumption expenditure and asset wealth at the village level. Models trained on panel data from five African countries between 2007 and 2021 explain up to 50% of spatial variation but struggle to explain temporal variation (R^2<0.04). This has important implications for retrospective policy evaluation, highlighting that in many contexts, non-traditional data sources cannot replace traditional survey data. Considering more nuanced input data constitutes a promising avenue for future research.