Thinking Like Transformers

Thinking like Transformers proposes a computational framework for Transformer-like calculations. The framework uses discrete computation to simulate Transformer computations. The resulting language RASP is a programming language where every program compiles down to a specific Transformer. In this blog post, we reimplement a variant of RASP in Python (RASPy). The language is roughly compatible with the original version, but with some syntactic changes for simplicity. With this language, we consider a challenging set of puzzles to walk through and understand how it works.