Structure-based Drug Design (SBDD), the task of designing 3D molecules (ligands) to bind with a target protein pocket, is a fundamental task in drug discovery. Recent geometric deep learning methods for SBDD fail to accurately generate valid docked structures without relying on physics-based post-processing (ie AutoDock Vina redocking), which resamples all the important geometric qualities of the molecule. Without 3D structure information or additional training on protein-ligand complexes as required by prior methods, EvoSBDD attains a state-of-the-art success rate of 86.4%, an average binding affinity of -10.27 kcal/mol, and demonstrates speed improvements up to 25.6x compared to the prior best method. EvoSBDD is the first method to maintain 100% generated molecule validity, novelty, and uniqueness and also excels in real-world off-target(s) binding prevention.