Abstract: This paper introduces a novel hybrid multi-start optimization strategy for solving inverse problems involving nonlinear dynamical systems and machine learning architectures, accelerated by GPU computing on both NVIDIA and AMD GPUs. The method combines Particle Swarm Optimization (PSO) and the Limited-memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) algorithms to address the challenges in parameter estimation for nonlinear dynamical systems. This hybrid strategy aims to leverage the global search capability of PSO and the efficient local convergence of L-BFGS. We experimentally show faster convergence by a factor of up to $8-30\times$ in a few non-convex problems with loss landscapes characterized by multiple local minima, which can cause regular optimization approaches to fail.