Skip to yearly menu bar Skip to main content


Poster
in
Workshop: Mathematical and Empirical Understanding of Foundation Models (ME-FoMo)

Towards Foundation Models with Mathematical Understanding

Peter Belcak · Roger Wattenhofer

Keywords: [ FACT ] [ integer sequences ] [ mathematical understanding ] [ mathematical concepts ] [ Transformer model ]


Abstract:

We investigate the ability of transformer models to build representations of integer sequences that are of utility to tasks where deeper mathematical understanding is needed.To that end, we train BERT-like transformer encoders to assess the impact of individual pre-training tasks on the quality of the resulting model, and evaluate them for sequence classification, continuation, unmasking, complexity prediction, and next sequence-part prediction.We find that the models both outperform benchmark baselines and provide reasonable estimates of the complexity of the mathematical rules behind the sequences.

Chat is not available.