In the fields of embodied AI and robotics, the training of visual policies via behavior cloning is a growing trend. While significant efforts have been directed toward refining model architectures and learning algorithms, this paper shifts the focus to the pivotal role of data in developing robust models. We propose a novel framework, termed AdaDemo (Adaptive Online Demonstration Expansion), which aims to improve multi-task policy learning by strategically expanding the demonstration dataset. This approach diverges from traditional methods by actively collecting new demonstrations in response to observed policy failures, particularly in multi-task settings. AdaDemo is driven by three core principles: prioritizing tasks with low policy performance, focusing on failed initial states for additional data collection, and adapting sampling strategies to emphasize challenging tasks. Through a comprehensive experimental setup involving two robotic manipulation benchmarks (RLBench and Adroit) across a total of 22 tasks, we demonstrate AdaDemo's effectiveness in creating high-quality multi-task demonstration datasets.