QLoRA: Fine-Tuning on Consumer Hardware

QLoRA (Quantized LoRA) extends LoRA to work with 4-bit quantized base models, making fine-tuning possible on a single consumer GPU.

How QLoRA Works

1. 4-bit NormalFloat (NF4): Quantize base model weights to 4-bit using information-theoretically optimal representation
2. Double Quantization: Quantize the quantization constants themselves (saves ~0.37 bits/parameter)
3. Paged Optimizers: Use NVIDIA unified memory to handle memory spikes during training
4. LoRA on top: Only train LoRA adapters in full precision (bfloat16)

Memory Requirements

Model	Full FT	LoRA	QLoRA
7B	112 GB	48 GB	6 GB
13B	208 GB	88 GB	10 GB
70B	1120 GB	360 GB	48 GB