From Foundation Model to Tiny Time Series Classifier through Knowledge Distillation

Time series foundation models provide strong generalization but remain computationally expensive for deployment in resource-constrained settings. We introduce TinyCN, a compact convolutional model trained via knowledge distillation from a transformer-based foundation model (Mantis-8M). Our training procedure transitions from representation alignment to task-specific optimization, enabling effective transfer of foundation representations into a lightweight CNN. Across all 128 UCR datasets, TinyCN achieves statistically significant improvements over Hybrid InceptionTime (HIT), the ensemble state-of-the-art, while being over $40\times$ smaller than Mantis and $10\times$ smaller than HIT. These results demonstrate that foundation representations can be effectively compressed into simple CNNs, achieving superior accuracy and efficiency for time series classification.