Skip to yearly menu bar Skip to main content


poster
in
Affinity Workshop: Tiny Papers Poster Session 3

BACKTRACKING MATHEMATICAL REASONING OF LANGUAGE MODELS TO THE PRETRAINING DATA

Yasaman Razeghi

#327
[ ] [ Project Page ]
Wed 8 May 1:45 a.m. PDT — 3:45 a.m. PDT

Abstract:

In this study, we identify subsets of model pretraining data that contribute to the math reasoning ability of language models and evaluate it on several mathematical tasks (e.g., addition, multiplication). We find that training on math-only data improves simple arithmetic but doesn't fully account for complex reasoning abilities, such as chain-of-thought reasoning. We also find that code data contributes to chain-of-thought reasoning while reducing arithmetic performance.

Live content is unavailable. Log in and register to view live content