To address the issues of catastrophic forgetting in Continual Generative Learning (CGL), dominant methods leverage the generative replay strategy. However, they often suffer from high time complexity and inferior generative sample quality. In this work, we develop an efficient and effective CGL method via Knowledge reconstruction and Feedback Consolidation (KFC). KFC extends the inherent data reconstruction properties of the variational autoencoder framework to historical knowledge reconstruction and re-encodes the current task's reconstructed data to the same posterior distribution as the original data. Experiments showcase that KFC achieves state-of-the-art performances in time complexity, sample quality, and accuracy on various CGL tasks. Code is available in Supplementary Materials.