Reinforcement learning plays an important role in various fields, and has fast development due to advancements in policy evaluation and learning methods, which enjoys advantages of large data size. However, when data are limited, directly applying evaluation methods does not necessarily result in a good policy evaluation. In this work we provide a framework to generate synthetic data with diffusion models, to enhance policy evaluation, which is supported by experiments.