Poster Thu, Apr 23, 2026 • 11:15 AM – 1:45 PM PDT Pavilion 3 P3-#1208

CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics

Weida Wang ⋅ Dongchen Huang ⋅ Jiatong LI ⋅ Tengchao Yang ⋅ Ziyang Zheng ⋅ Chuyi Peng ⋅ Di Zhang ⋅ Dong Han ⋅ Benteng Chen ⋅ Binzhao Luo ⋅ Zhiyu Liu ⋅ kunling liu ⋅ Zhiyuan Gao ⋅ Shiqigeng geng ⋅ Wei Ma ⋅ Jiaming Su ⋅ Xin Li ⋅ Shuchen Pu ⋅ Yuhan Shui ⋅ Qianjia Cheng ⋅ Zhihao Dou ⋅ Dongfei Cui ⋅ Changyong He ⋅ Jin Zeng ⋅ Zeke Xie ⋅ Mao Su ⋅ Dongzhan Zhou ⋅ Yuqiang Li ⋅ Wanli Ouyang ⋅ Yunqi Cai ⋅ Xi Dai ⋅ Shufei Zhang ⋅ LEI BAI ⋅ Jinguang Cheng ⋅ Zhong Fang ⋅ Hongming Weng

Project Page

Abstract

We introduce CMPhysBench, designed to assess the proficiency of Large Language Models (LLMs) in Condensed Matter Physics, as a novel Benchmark. CMPhysBench is composed of more than 520 graduate-level meticulously curated questions covering both representative subfields and foundational theoretical frameworks of condensed matter physics, such as magnetism, superconductivity, strongly correlated systems, etc. To ensure a deep understanding of the problem-solving process,we focus exclusively on calculation problems, requiring LLMs to independently generate comprehensive solutions. Meanwhile, leveraging tree-based representations of expressions, we introduce the Scalable Expression Edit Distance (SEED) score, which provides fine-grained (non-binary) partial credit and yields a more accurate assessment of similarity between prediction and ground-truth. Our results show that even the best models, Grok-4, reach only 36 average SEED score and 29% accuracy on CMPhysBench, underscoring a significant capability gap, especially for this practical and frontier domain relative to traditional physics.