Sparsity for Communication-Efficient LoRA

Recently, several works have used unstructured pruning to augment adapter methods. However, these ``sparse adapter'' methods have limited communication benefits in federated learning. In this work, we propose a simple baseline which combines LoRA with a constant sparsity during communication only. On three FL image and text tasks, our method reduces communication costs by up to $10\times$ over vanilla (dense) LoRA and up to $5\times$ over more complex sparse LoRA baselines. Our work highlights the importance of considering system-specific constraints when developing efficient fine-tuning approaches, and serves as a competitive baseline for future work in federated fine-tuning.