Eliciting Human Preferences with Language Models

Language models (LMs) can be directed to perform user- and context-dependenttasks by using labeled examples or natural language prompts.But selecting examples or writing prompts can be challenging---especially in tasks that require users to precisely articulate nebulous preferences or reason about complex edge cases. For such tasks, we introduce Generative Active Task Elicitation (GATE), a method for using LMs themselves to guide the task specification process. GATE is a learning framework in which models elicit and infer human preferences through free-form, language-based interaction with users.We identify prototypical challenges that users face when specifying preferences, and design three preference modeling tasks to study these challenges:content recommendation, moral reasoning, and email validation.In preregistered experiments, we show that LMs that learn to perform these tasks using GATE (by interactively querying users with open-ended questions) obtain preference specifications that are more informative than user-written prompts or examples. GATE matches existing task specification methods in the moral reasoning task, and significantly outperforms them in the content recommendation and email validation tasks. Users additionally report that interactive task elicitation requires less effort than prompting or example labeling and surfaces considerations that they did not anticipate on their own. Our findings suggest that LM-driven elicitation can be a powerful tool for aligning models to complex human preferences and values.