Enforcing Temporal Constraints for LLM Agents

We present Agent-C, a novel framework that provides run-time guarantees ensuring LLM agents adhere to temporal and state-dependent constraints. Agent-C introduces a domain-specific language for expressing temporal properties, translates specifications to first-order logic, and interleaves SMT solving with constrained generation techniques to detect non-compliant agent actions during token generation. If the agent generates a non-compliant action, Agent-C backtracks the generation and samples a compliant alternative. We evaluate Agent-C on two real-world applications: retail customer service and airline ticket reservation, with multiple language models (open-weight and closed-source) as agents. Our results demonstrate that Agent-C achieves perfect safety (100% conformance, 0% harm) in both benign and adversarial scenarios, while improving task utility compared to state-of-the-art guardrails and unrestricted agents. On state-of-the-art closed-source models, Agent-C improves conformance (from 77.4% to 100% for Claude Sonnet 4.5 and 83.7% to 100% for GPT-5), while increasing utility (from 66.1% to 69.3% and 61.0% to 64.3%, respectively), setting a new state-of-the-art frontier for reliable agentic systems. More details about the framework (e.g., code repository, demo) can be found on the project website: https://go.illinois.edu/agentc.