Skip to yearly menu bar Skip to main content


Virtual presentation / top 25% paper

STREET: A MULTI-TASK STRUCTURED REASONING AND EXPLANATION BENCHMARK

Danilo Neves Ribeiro · Shen Wang · Xiaofei Ma · Henghui Zhu · Rui Dong · Deguang Kong · Juliette Burger · Anjelica Ramos · zhiheng huang · William Wang · George Karypis · Bing Xiang · Dan Roth

Keywords: [ dataset ] [ natural language understanding ] [ question answering ] [ soft reasoning ] [ structured explanations ] [ Infrastructure ]


Abstract:

We introduce STREET, a unified multi-task and multi-domain natural language reasoning and explanation benchmark. Unlike most existing question-answering (QA) datasets, we expect models to not only answer questions, but also produce step-by-step structured explanations describing how premises in the question are used to produce intermediate conclusions that can prove the correctness of a certain answer. We perform extensive evaluation with popular language models such as few-shot prompting GPT-3 and fine-tuned T5. We find that these models still lag behind human performance when producing such structured reasoning steps. We believe this work will provide a way for the community to better train and test systems on multi-step reasoning and explanations in natural language.

Chat is not available.