Skip to yearly menu bar Skip to main content


Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning

Haowen Wang · Tao Sun · Congyun Jin · Yingbo Wang · Yibo Fan · Yunqi Xu · Yuliang Du · Cong Fan

Halle B #278
[ ]
Thu 9 May 7:30 a.m. PDT — 9:30 a.m. PDT

Abstract: Modular and composable transfer learning is an emerging direction in the field of Parameter Efficient Fine-Tuning, as it enables neural networks to better organize various aspects of knowledge, leading to improved cross-task generalization.In this paper, we introduce a novel approach Customized Polytropon ($\texttt{C-Poly}$) that combines task-common skills and task-specific skills, while the skill parameters being highly parameterized using low-rank techniques.Each task is associated with a customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super-NaturalInstructions and the SuperGLUE benchmarks.Our findings demonstrate that $\texttt{C-Poly}$ outperforms fully-shared, task-specific, and skill-indistinguishable baselines, significantly enhancing the sample efficiency in multi-task learning scenarios.

Chat is not available.