Poster

GneissWeb: Preparing High Quality Data for LLMs at Scale

Hajar Emami Gohari · Swanand Kadhe · Yousaf Shah · Constantin Adam · Abdulhamid Adebayo · Praneet Adusumilli · Farhan Ahmed · Nathalie Baracaldo · Santosh Borse · Yuan-Chi Chang · Xuan-Hong Dang · Nirmit Desai · Revital Eres · Ran Iwamoto · Alexei Karve · Yan Koyfman · Wei-Han Lee · Changchang Liu · Boris Lublinsky · Takuya Ohko · Pablo Pesce · Maroun Touma · Shiqiang Wang · Shalisha Witherspooon · Herbert Woisetschläger · David Wood · Kun-Lung Wu · Issei Yoshida · Syed Zawad · Petros Zerfos · Yi Zhou · Bishwaranjan Bhattacharjee

Abstract

Data quantity and quality play a vital role in determining the performance of Large Language Models (LLMs). High-quality data, in particular, can significantly boost the LLM's ability to generalize on a wide range of downstream tasks. In this paper, we introduce GneissWeb, a large dataset of around 10 trillion tokens that caters to the data quality and quantity requirements of training LLMs. Our GneissWeb recipe that produced the dataset consists of sharded exact sub-string deduplication and a judiciously constructed ensemble of quality filters. GneissWeb goes beyond simple model-based quality filtering used in recent datasets by designing an ensemble of filters incorporating novel quality filters. Novel components enable us to achieve a favorable trade-off between data quality and quantity, producing models that outperform models trained on state-of-the-art open large datasets (5+ trillion tokens). We show that models trained using GneissWeb outperform those trained on FineWeb-V1.1.0 by 2.73 percentage points in terms of average scores on a set of 11 commonly used benchmarks (both zero-shot and few-shot) for pre-training dataset evaluation. When the evaluation set is extended to 20 benchmarks (both zero-shot and few-shot), models trained using GneissWeb still achieve a 1.75 percentage points gain over those trained on FineWeb-V1.1.0.