A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control

Developing transformer-based models in online reinforcement learning (RL) faces a wide range of difficulties such as training instability or suboptimal behavior. In this paper, we find out whether the transformer architecture can be considered as a backbone for RL algorithms. We show that transformers can be trained by classical online RL algorithms without requiring global changes in the training process. Moreover, we explore different transformer architectures and ways to train them. As a result we form a set of recommendations and practical takeaways about how to develop stable approaches of transformer training. We hope that our work will help in understanding the intricacies of configuring transformers for reinforcement learning and will allow to formulate the basic principles of forming a training pipeline for transformer-based architectures.