Designing protein sequences that adopt a specific three-dimensional (3D) structure remains a significant challenge in computational biology. Existing methods often rely on static learning approaches that, during inference, cannot dynamically integrate rich multimodal representations from larger datasets, specifically the combined information of 3D structure and 1D sequence. In this paper, weintroduce PRISM, a novel retrieval-augmented generation (RAG) framework that enhances protein sequence design by dynamically incorporating fine-grained multimodal representations from a larger set of known structure-sequence pairs. Our experiments demonstrate that PRISM significantly outperforms state-of-the-art techniques in sequence recovery, emphasizing the advantages of incorporating fine-grained, multimodal retrieval-augmented generation in protein design.