Optimizing Diffusion ODE Solvers via Within-Step Parallelism and Order Composition

Diffusion model inference can be formulated as the integration of a continuous-time ODE, enabling the use of numerical solvers that efficiently reduce the number of denoising steps. Nevertheless, repeated neural network evaluations remain a major computational bottleneck. Most existing diffusion solvers apply the same numerical integration method across time and rely on sequential function evaluations within each solver step. We explore several solver modifications aimed at improving inference efficiency by relaxing these design choices. First, we study within-step parallelization by relaxing dependencies between function evaluations inside a single step of explicit Runge–Kutta solvers. By reusing a shared state across these evaluations, this approach enables parallel execution within a step at the cost of reduced formal order. We include complementary analyses of parallel exponential solvers in the appendix. Further, we observe that diffusion trajectories evolve unevenly over time and often contain extended regions with limited change. Motivated by this behavior, we propose order composition, which mixes solver orders over time by applying low-order methods in slowly varying regions and higher-order methods only where the trajectory changes rapidly. Together, these findings indicate that carefully exploiting trajectory structure can reduce the computational cost of diffusion inference. Finally, we implemented all proposed solvers within the Diffusers library.