Explaining multivariate time series is a compound challenge, as it requires identifying important locations in the time series and matching complex temporal patterns.Although previous saliency-based methods addressed the challenges,their perturbation may not alleviate the distribution shift issue, which is inevitable especially in heterogeneous samples.We present ContraLSP, a locally sparse model that introduces counterfactual samples to build uninformative perturbations but keeps distribution using contrastive learning.Furthermore, we incorporate sample-specific sparse gates to generate more binary-skewed and smooth masks, which easily integrate temporal trends and select the salient features parsimoniously.Empirical studies on both synthetic and real-world datasets show that ContraLSP outperforms state-of-the-art models, demonstrating a substantial improvement in explanation quality for time series data.The source code is available at \url{https://github.com/zichuan-liu/ContraLSP}.