Understanding in-context learning in transformers
Simone Rossi · Rui Yuan · Thomas Hannagan
2024 Poster
in
Affinity Event: Blog Track Session 8
in
Affinity Event: Blog Track Session 8
Abstract
We propose a critical review on the phenomenon of In-Context Learning (ICL) in transformer architectures. Focusing on the article Transformers Learn In-Context by Gradient Descent by J. von Oswald et al., published in ICML 2023 earlier this year, we provide detailed explanations and illustrations of the mechanisms involved. We also contribute novel analyses on ICL, discuss recent developments and we point to open questions in this area of research.
Video
Chat is not available.
Successful Page Load