Don't Pay Attention to the Noise: Learning Self-supervised Light Curve Representations with a Denoising Time Series Transformer

Astrophysical light curves are particularly challenging data objects due to theintensity and variety of noise contaminating them. Yet, despite the astronomi-cal volumes of light curves available, the majority of algorithms used to processthem are still operating on a per-sample basis. To remedy this, we propose a sim-ple Transformer model –called Denoising Time Series Transformer (DTST)– andshow that it excels at removing the noise in datasets of time series when trainedwith a masked objective, even when no clean targets are available. Moreover, theuse of self-attention enables rich and illustrative queries into the learned represen-tations. We present experiments on real stellar light curves from the TransitingExoplanet Space Satellite (TESS), showing advantages of our approach comparedto traditional denoising techniques.