Low-rank Adaptation (LoRA)

LoRA is a training methodology designed to speed up the training process of large models while using less memory. By introducing pairs of rank-decomposition weight matrices (known as update matrices) to existing weights and solely training these added matrices, LoRA maintains the efficacy of large models without the extensive computational requirements traditionally associated with them.

How LoRA Works

1. Addition of Update Matrices: Instead of retraining the entire model, LoRA introduces pairs of rank-decomposition weight matrices to the existing pretrained weights of the model. These matrices are called update matrices.
2. Frozen Pretrained Weights: One of LoRA's distinct advantages is that it keeps the pretrained weights unaltered during the training process. This approach helps in preventing the phenomenon called catastrophic forgetting, where the model forgets previously learned knowledge.
3. Memory Efficiency: The rank-decomposition matrices introduced by LoRA have far fewer parameters compared to the original model. This makes the newly trained weights (LoRA weights) lightweight and easily transferable.