In the pursuit of novel drug molecules, the optimization stage for enhanced safety, efficacy, and pharmacokinetics presents a significant challenge. Deep generative models, particularly diffusion models, emerge as an effective strategy for this purpose. Our method integrates 3D protein structures with diffusion models, facilitating the generation of new ligands that consider both the molecular scaffold and the specific environment of the target protein's binding pocket. As a demonstration of its effectiveness, we applied this proposed approach to generate new ligands targeting the colony-stimulating factor 1 receptor. The outcomes highlight the method's ability to design potent inhibitors, achieving enhanced inhibitory efficacy in compared to existing inhibitors, as confirmed by in vitro assays.