MATRIX: Stress-Testing LLM Reasoning in Materials Science

Large Language Models (LLMs) have demonstrated remarkable abilities in scientific reasoning, yet their reasoning capabilities in materials science remain underexplored. To fill this gap, we introduce MATRIX, a comprehensive materials science reasoning and intelligence examination comprising 1,340 problems that span the essential subdisciplines of materials science. MATRIX features a structured and fine-grained taxonomy that categorizes materials science questions into 6 primary fields and 31 sub-fields, and includes a three-tier difficulty classification based on the reasoning length required to solve each question. MATRIX provides detailed reference solutions enabling precise error analysis and incorporates multimodal reasoning through visual contexts in 315 questions. Evaluations of leading models reveal that even the highest-performing model, Gemini-2.5-Pro, achieves under 80% accuracy on college-level materials science questions, highlighting the complexity of MATRIX. Our systematic analysis of different reasoning strategies—basic chain-of-thought, tool augmentation, and self-correction—demonstrates that no single method consistently excels across all scenarios. We further analyze performance by difficulty level, examine trade-offs between efficiency and accuracy, highlight the challenges inherent in multimodal reasoning tasks, analyze failure modes across LLMs and reasoning methods, and evaluate the influence of retrieval-augmented generation. MATRIX thus establishes a comprehensive and solid benchmark for assessing and driving improvements in the scientific reasoning capabilities of LLMs within the materials science domain.