Low-Rank Adaptation (LoRA) freezes pre-trained model weights $W_0 \in \mathbb{R}^{d \times k}$ and injects trainable rank decomposition matrices into each layer of the Transformer architecture:
$$W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} (B \cdot A)$$
$$\text{where } B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, \quad r \ll \min(d, k)$$
Key Architectural Properties:
- Initialization: Matrix $A$ is initialized from a Gaussian distribution $\mathcal{N}(0, \sigma^2)$, while matrix $B$ is initialized to 0. Consequently, $\Delta W = B \cdot A = 0$ at the start of training, preserving the exact original model behavior.
- Scaling Factor $\alpha$: The constant hyperparameter $\alpha$ stabilizes training when varying the rank $r$.
- Zero Inference Latency: For production deployment, weights can be merged directly into the base weights prior to serving: $W_{\text{merged}} = W_0 + \frac{\alpha}{r}(BA)$.
PyTorch LoRA Implementation:
import torch
import torch.nn as nn
class LoRALinear(nn.Module):
def __init__(self, in_features: int, out_features: int, rank: int = 8, alpha: float = 16.0):
super().__init__()
self.linear = nn.Linear(in_features, out_features, bias=False)
self.linear.weight.requires_grad = False
self.rank = rank
self.scaling = alpha / rank
self.lora_A = nn.Parameter(torch.randn(rank, in_features) * 0.02)
self.lora_B = nn.Parameter(torch.zeros(out_features, rank))
def forward(self, x: torch.Tensor) -> torch.Tensor:
base_out = self.linear(x)
lora_out = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling
return base_out + lora_out
