While the recent advances in automatic emotion recognition with human physical signals such as audio, visual and textual inputs have been remarkable, research on emotion recognition with internal physiological signals has received considerable attention only in recent years, and most of the studies focused on feature engineering and traditional machine learning algorithms. In this study, we propose an advanced domain alignment transformer (DATransformer) framework that addresses the major challenges of physiological signals based emotion recognition--the domain inconsistency and sample rate difference between the multivariate physiological signals and emotional states. Our proposed DATransformer framework does not require any preprocessing on the raw physiological signal inputs, but can obtain comparable or even better emotion recognition performance than the preprocessed signals. We evaluate the proposed DATransformer on the Continuously Annotated Signals of Emotion (CASE) dataset and achieve the state-of-the-art (SOTA) performance.