Fast and accurate generation of molecular conformers is desired for downstream computational chemistry and drug discovery tasks. In this work, we propose two mechanisms for accelerating the training and inference of flow-based generative model for 3D molecular conformer generation. For fast training, we introduce the SO(3)-Averaged Flow training objective, which we show to converge faster and generate better conformer ensembles compared to conditional optimal transport and Kabsch alignment-based optimal transport flow. For fast inference, we demonstrate that reflow methods and distillation of these models enable few-steps or even one-step molecular conformer generation with high quality. Using these two techniques, we demonstrate a model that can match the performance of strong transformer baselines with only a fraction of the number of parameters and generation steps.