One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
Pre-Print
Arnav Chavan1,2*, Zhuang Liu3, Deepak Gupta2, Eric Xing1,4, Zhiqiang Shen1*
1MBZUAI, 2Transmute AI Lab, 3FAIR, Meta 4CMU (*equal contribution)
Abstract
We present Generalized LoRA (GLoRA), an advanced approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, layer-wise structure search that learns individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adjusts to new tasks through additional dimensions on weights and activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured benchmarks, achieving superior accuracy with fewer parameters and computations on various datasets. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications.
Schematic representation of a linear layer adapted with GLoRA. W₀ and b₀ are frozen weight and bias respectively. A, B, C, D and E are trainable support tensors of GLoRA.
Simple Forward Pass During Fine-Tuning:
def forward(self, input)
optimal_weight = self.weight + self.weight*A + B
optimal_bias = self.bias + self.bias*D + E
optimal_prompt = torch.matmul(self.weight, C).squeeze()
return F.linear(input, optimal_weight, optimal_bias+optimal_prompt)
BibTeX