The variability of solar photovoltaic (PV) output, particularly that caused by rapidly changing cloud dynamics, challenges the reliability of renewable energy systems. Solar forecasting based on cloud observations collected by ground-level sky cameras shows promising performance in anticipating short-term solar power fluctuations. However, current deep learning methods often rely on a single dataset with limited sample diversity for training, and thus generalize poorly to new locations and different sky conditions. Moreover, the lack of a standardized dataset hinders the consistent comparison of existing solar forecasting methods. To close these gaps, we propose to build a large-scale standardized sky image dataset --- SkyImageNet --- by assembling, harmonizing, and processing suitable open-source datasets collected in various geographical locations. An accompanying python package will be developed to streamline the process of utilizing SkyImageNet in a machine learning framework. We hope that the outcomes of this project will foster the development of more robust forecasting systems, advance the comparability of short-term solar forecasting model performances, and further facilitate the transition to the next generation of sustainable energy systems.