Price-based demand response management (DR) enables households to provide the flexibility required in power grids with a high share of volatile renewable energy sources. Multi-agent reinforcement learning (MARL) offers a powerful, decentralized decision-making tool for autonomous agents participating in DR programs. Unfortunately, MARL algorithms do not naturally allow incorporating safety guarantees, preventing their real-world deployment. To meet safety constraints, we propose a safety layer which minimally adjusts each agent’s decisions. We investigate the influence of incentivizing the agents to minimize safety constraint violation by adding a scalar safety feedback to the reward. Results show that using the feedback during training improves both convergence speed and performance.