Learning Transferable Latent User Preferences for Human-Aligned Decision Making

Large language models (LLMs) are increasingly used as reasoning modules for autonomous agents. While they are efficient in certain tasks, LLMs often struggle to produce human-aligned solutions. Human-aligned decision making requires accounting for both explicitly stated goals and latent user preferences that shape how ambiguous situations should be resolved. Existing approaches to incorporating such preferences either rely on extensive and repeated user interactions or fail to generalize latent preferences across tasks and contexts, limiting their practical applicability. We consider a setting in which an agent uses an LLM for high-level reasoning and as an intermediary between the human and the agent. In this role, the LLM is responsible for inferring latent user preferences from limited interactions, which guides downstream decision making. We introduce CLIPR (Conversational Learning for Inferring Preferences and Reasoning), a framework that learns actionable, transferable representations of latent user preferences from minimal conversational input and applies them to both in-distribution and out-of-distribution ambiguous tasks across multiple environments. Evaluations on three datasets show that CLIPR consistently outperforms existing methods in aligning agent behavior with user preferences, while significantly reducing training, inference, and runtime costs.