Prompt Optimization with Logged Bandit Data

We study how to use naturally available user feedback, such as clicks, to optimize a prompt policy for generating sentences with large language models (LLMs). Naive approaches, including regression-based and importance sampling-based ones, suffer either from biased log data or variance caused by the large action space of prompt. To circumvent these challenges, we propose a way to leverage similarity and smoothness in the (generated) sentence embedding space, substantially reducing variance in the policy gradients while maintaining a small bias. Initial experiments on synthetic data demonstrate the effectiveness of our approach. We also plan to publish the extended benchmark and simulator as open-source software.