Abstract: Neural ODEs have exhibited remarkable capabilities in continuously modeling dynamical systems from observational data. However, existing training methods, often based on adaptive-step-size numerical ODE solvers, are time-consuming and may introduce additional errors. Despite recent attempts to address these issues, existing methods rely heavily on numerical ODE solvers and lack efficient solutions. In this work, we propose the Fast-VF Neural ODE, a novel approach based on variational formulation (VF) to accelerate the training of Neural ODEs for dynamical systems. To further mitigate the influence of oscillatory terms in the VF loss, we incorporate the Filon's method into our design. Extensive experimental results show that our method can accelerate the training of Neural ODEs by 10 $\times$ to 100 $\times$ compared to the baselines while achieving comparable accuracy in irregularly-sampled dynamical systems.