Generating CAD Code with Vision-Language Models for 3D Designs

Generative AI has transformed the fields of Design and Manufacturing by providingefficient and automated methods for generating and modifying 3D objects. Oneapproach involves using Large Language Models (LLMs) to generate Computer-Aided Design (CAD) scripting code, which can then be executed to render a 3Dobject; however, the resulting 3D object may not meet the specified requirements.Testing the correctness of CAD generated code is challenging due to the complexityand structure of 3D objects (e.g., shapes, surfaces, and dimensions) that are notfeasible in code. In this paper, we introduce CADCodeVerify, a novel approach toiteratively verify and improve 3D objects generated from CAD code. Our approachworks by producing ameliorative feedback by prompting a Vision-Language Model(VLM) to generate and answer a set of validation questions to verify the generatedobject and prompt the VLM to correct deviations. To evaluate CADCodeVerify, weintroduce, CADPrompt, the first benchmark for CAD code generation, consisting of200 natural language prompts paired with expert-annotated scripting code for 3Dobjects to benchmark progress. Our findings show that CADCodeVerify improvesVLM performance by providing visual feedback, enhancing the structure of the 3Dobjects, and increasing the success rate of the compiled program. When applied toGPT-4, CADCodeVerify achieved a 7.30% reduction in Point Cloud distance and a5.0% improvement in success rate compared to prior work.