Skip to yearly menu bar Skip to main content


Poster

Language Models Are Implicitly Continuous

Samuele Marro · Davide Evangelista · X. Huang · Emanuele La Malfa · Michele Lombardi · Michael Wooldridge

Hall 3 + Hall 2B #250
[ ]
Thu 24 Apr 7 p.m. PDT — 9:30 p.m. PDT

Abstract:

Language is typically modelled with discrete sequences. However, the most successful approaches to language modelling, namely neural networks, are continuous and smooth function approximators.In this work, we show that Transformer-based language models implicitly learn to represent sentences as continuous-time functions defined over a continuous input space. This phenomenon occurs in most state-of-the-art Large Language Models (LLMs), including Llama2, Llama3, Phi3, Gemma, Gemma2, and Mistral, and suggests that LLMs reason about language in ways that fundamentally differ from humans.Our work formally extends Transformers to capture the nuances of time and space continuity in both input and output space.Our results challenge the traditional interpretation of how LLMs understand language, with several linguistic and engineering implications.

Live content is unavailable. Log in and register to view live content