Recent advancements in generative models for Structure-based Drug Design (SBDD) have surpassed traditional methods, but their confined scope restricts the data available for training and their practical applications. To overcome these limitations, we introduce a flexible SBDD method based on an equivariant diffusion model, which was trained via a broadly applicable training objective and could therefore leverage the large and diverse sets of protein-ligand complexes available. Our approach excels in a wide range of SBDD subtasks, including scaffold hopping, fragment merging, and fragment growing, without requiring specialized training. Additionally, it not only generates hits but can also optimize desirable properties of existing hits, such as binding score and synthetic accessibility. Our optimization framework opens up new opportunities for negative design and increasing target specificity. It can be utilized in both a highly automated and manually controlled manner, offering drug discovery scientists fine-grained control. This versatile method has the potential to be valuable for a broad range of molecular design tasks, serving as a foundation for future advancements in the field.