Learning Safe Robot Planning from Unsafe Experiences: An Episodic Memory Approach for LLM-based Agents

LLM-based robotic agents can generate unsafe commands that harm humans, objects, or the environment. We propose an episodic safety memory system that learns to filter harmful instructions by storing and retrieving past violation experiences. Our memory architecture maintains episodic stores of unsafe instances and consolidates recurring patterns into semantic constraints. Real-time memory retrieval blocks similar unsafe commands before execution. Preliminary experiments on ConceptGraphs-based planning show 94% safety rate (vs. 52% baseline) while maintaining 97% task success, suggesting that learning from unsafe experiences can enable safer LLM-based robotic agents.