We propose two extensions to Robust Adversarial Reinforcement Learning. (Pinto et al., 2017) One is to add a penalty that brings the training domain closer to the test domain to the objective function of the adversarial agent. The other method trains multiple adversarial agents for one protagonist. We conducted experiments with the physical simulator benchmark task. The results show that our method improves performance in the test domain compared to the baseline.