Neural forecasting has become an active research area, with advancements in architectural design steadily improving performance and scalability. Most existing approaches produce forecasts using a fixed parametric function with historical values as inputs. We identify performance limitations of this approach in handling two recurrent challenges in IoT data: distribution shifts and missing data. We propose SpectraNet, a model based on a new paradigm for time-series forecasting. We introduce a latent factor inference method that matches the model's output on past observations. Theoretically motivated as a MAP estimation of the posterior distribution of latent factors, the inference process provides additional flexibility to adjust forecasts based on the latest information. We identify three advantages of our method: (i) SoTA performance with 92% fewer parameters and similar training times; (ii) superior robustness to missing data and distribution shifts; and (iii) capability to simultaneously produce forecasts and interpolate past missing data, unifying imputation and forecasting tasks.