Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaption

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient method for fine-tuning Large Langauge Models. The weight matrix is updated as $W=W_0+sBA$, where $W_0$ is the original frozen weight, $s$ is a scaling factor and $A$,$B$ are trainable low-rank matrices. In this paper, we first theoretically show that, LoRA can naturally achieve and sustain stable feature learning (i.e., can be self-stabilized) given appropriate hyper-parameters and initializations of $A$ and $B$. However, we also claim that the non-zero initialization of $A$ could potentially compromise self-stability. To address this issue, we propose Stable-LoRA, a weight-shrinkage optimization strategy that enhances stability of LoRA feature learning. By progressively shrinking $A$ in the earliest training steps, Stable-LoRA is theoretically proved and empirically verified to prevent potential instability of LoRA while preserving the benefits of the non-zero start. With only 3 lines of code modification, Stable-LoRA consistently outperforms classical LoRA and other baselines in accuracies across various tasks, with no extra memory usage and negligible additional computation costs.