We present on-going work toward building Stasis, a suite of reinforcement learning (RL) environments that aim to maintain realism for human-centric agents operating in real-world settings. Through representation learning and alignment with real-world offline data, Stasis allows for the evaluation of RL algorithms in offline environments with adjustable characteristics, such as observability, heterogeneity and levels of missing data. We aim to introduce environments the encourage training RL agents that are capable of maintaining a level of performance and robustness comparable to agents trained in real-world online environments, while avoiding the high cost and risks associated with making mistakes during online training. We provide examples of two environments that will be part of Stasis and discuss its implications for the deployment of RL-based systems in sensitive and high-risk areas of application.