Accurate mesh-based simulation is central to modeling phenomena governed by PDEs, such as flow, elasticity, and climate. Recent machine learning solutions, including Graph Neural Networks (GNNs) and Fourier Neural Operators (FNOs), enable faster approximations but can struggle with long-range interactions, irregular mesh topologies, or fixed time steps. To address the above challenges, we introduce SpectralFlowNet, a unified framework for mesh-based PDE simulation that marries graph spectral methods with continuous-time neural dynamics. By projecting mesh data onto an intrinsic spectral basis via the Graph Fourier Transform (GFT) and evolving these spectral coefficients using Neural Ordinary Differential Equations (ODEs), our model naturally handles multiscale spatial structures and temporal dynamics. This resolution-invariant, multiscale approach achieves state-of-the-art performance on plastic deformation tasks and demonstrates robust zero-shot transfer across resolutions.