Amortized simulator-based inference has emerged as a powerful framework for tackling inverse problems and Bayesian inference in many computational sciences by learning the reverse mapping from observed data to parameters. Once trained on many simulated parameter-data pairs, these methods afford parameter inference for any particular dataset, yielding high-quality posterior samples with only one or a few forward passes of a neural network. While amortized methods offer significant advantages in terms of efficiency and reusability across datasets, they are typically constrained by their training conditions -- particularly the prior distribution of parameters used during training. In this paper, we introduce PriorGuide, a technique that enables on-the-fly adaptation to arbitrary priors at inference time for diffusion-based amortized inference methods. Our technique allows users to incorporate new information or expert knowledge at runtime without costly retraining.