Skip to yearly menu bar Skip to main content


Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models

Sheng Shen · Le Hou · Yanqi Zhou · Nan Du · Shayne Longpre · Jason Wei · Hyung Won Chung · Barret Zoph · William Fedus · Xinyun Chen · Tu Vu · Yuexin Wu · Wuyang Chen · Albert Webson · Yunxuan Li · Vincent Zhao · Hongkun Yu · Kurt Keutzer · trevor darrell · Denny Zhou

Halle B #84
[ ]
Wed 8 May 7:30 a.m. PDT — 9:30 a.m. PDT


Sparse Mixture-of-Experts (MoE) is a neural architecture design that adds learnable parameters to Large Language Models (LLMs) without increasing computational complexity (FLOPs). Instruction tuning is a technique for training LLMs to follow instructions. We advocate combining these two approaches, as we find that MoE models benefit more from instruction tuning than dense models. In particular, we conduct empirical studies across three experimental setups: (i) Direct finetuning on individual downstream tasks devoid of instruction tuning; (ii) Instruction tuning followed by in-context few-shot or zero-shot generalization on downstream tasks; and (iii) Instruction tuning supplemented by further finetuning on individual downstream tasks. In the first scenario, MoE models overall underperform dense models of identical computational capacity. This narrative, however, dramatically changes with the introduction of instruction tuning (in the second and third scenarios), used independently or in conjunction with task-specific finetuning. Our most powerful model, FLAN-MoE-32B, surpasses the performance of Flan-PaLM-62B on four benchmark tasks, while using only a third of the FLOPs. The advancements embodied by FLAN-MoE inspire a reevaluation of the design principles of large-scale, high-performance language models in the framework of task-agnostic learning.

Chat is not available.