← All papers / 2023

QLoRA: Efficient Finetuning of Quantized LLMs

2023 Efficient and edge inference
Download PDF

QLoRA: Efficient Finetuning of Quantized LLMs

Metadata

Compute Setup

The paper explicitly targets single-GPU finetuning of very large language models. It states that QLoRA reduces memory enough to finetune a 65B-parameter model on a single 48GB GPU while preserving full 16-bit finetuning performance. It also states that 33B models can be trained on a single 24GB consumer GPU, and that the best 33B Guanaco model can be trained in less than 12 hours. For the largest model, the paper reports reaching 99.3% of ChatGPT's Vicuna benchmark performance using a single professional GPU over 24 hours.

The baseline memory problem is severe: regular 16-bit finetuning of LLaMA 65B requires more than 780GB of GPU memory. QLoRA reduces that to less than 48GB for finetuning and reports Guanaco 65B inference/training memory entries around 41GB in the Vicuna table. Guanaco 33B is listed at 21GB, Guanaco 13B at 10GB, and Guanaco 7B at 5GB. The paper also uses NVIDIA unified memory through paged optimizers, moving optimizer-state pages between CPU RAM and GPU memory when transient spikes would otherwise cause out-of-memory failures.

Bottleneck

The bottleneck is GPU memory, not only arithmetic throughput. LoRA already reduces trainable parameters, but in large-model finetuning the frozen base still dominates memory. Activations, input gradients, optimizer states, quantization metadata, and gradient-checkpointing spikes all compete for the same GPU memory. A 65B model in 16-bit form is far beyond a single 48GB card.

The bound is not merely the 65B parameter count. BF16 storage for the frozen LLaMA 65B base would be about 130 GB before activations or optimizer state, while regular 16-bit finetuning is reported above 780 GB because gradients, optimizer states, trainable weights, activations, and transient checkpointing peaks all count. QLoRA changes the bound by storing the frozen base in 4-bit form, leaving LoRA adapters as the trainable path, and using paged optimizers for spikes that would otherwise exceed the 24GB or 48GB device limit.

The paper emphasizes that prior quantization methods were mostly inference tools. Training through a quantized model is harder because gradients must flow through the frozen base into adapters without destroying quality. Long sequence lengths and larger minibatches create temporary memory spikes during checkpointing; these spikes can be enough to break an otherwise fitting configuration. The challenge is therefore to make the base model small in memory, keep adapter training accurate, and handle rare peak-memory events without slowing the whole run too much.

Method Adaptation

QLoRA adapts parameter-efficient finetuning around a frozen 4-bit base model. The base weights are stored in 4-bit NormalFloat (NF4), a datatype designed for normally distributed weights. During computation, weights are dequantized to the 16-bit computation datatype, and gradients are backpropagated through the frozen quantized model into LoRA adapters. The adapters remain the trainable path; the quantized base supplies the large pretrained representation without requiring high-precision storage.

Double quantization attacks metadata overhead. Small quantization block sizes improve precision, but each block needs quantization constants. With 32-bit constants and block size 64, constants alone cost 0.5 bits per parameter. QLoRA quantizes those constants too, reducing the average metadata cost to 0.127 bits per parameter, a savings of 0.373 bits per parameter, or roughly 3GB for a 65B model.

Paged optimizers address peak memory rather than average memory. They allocate optimizer states in NVIDIA unified memory and let pages move between CPU and GPU when the GPU runs out of memory. This is specifically designed for gradient-checkpointing spikes with long sequences. The paper reports that paged optimizers are critical for 33B/65B tuning on single 24GB/48GB GPUs, while also measuring 65B runs on 48GB GPUs where batch size 16 gives the same training speed as regular optimizers.

Evidence

The paper reports the main memory result directly: 65B finetuning falls from more than 780GB under regular 16-bit finetuning to less than 48GB with QLoRA. The memory table for chatbot models lists Guanaco 65B at 41GB and Guanaco 33B at 21GB. The appendix memory breakdown shows LLaMA base footprints of about 6.9GB for 7B, 11.3GB for 13B, 24.7GB for 33B, and 45.0GB for 65B under the QLoRA setup, with paged optimizers needed to make tight cases fit when activations spike.

Quality evidence comes from both academic and chatbot evaluations. Across GLUE, Super-NaturalInstructions, and MMLU adapter comparisons, the paper reports that 4-bit QLoRA with NF4 and double quantization matches 16-bit full finetuning or 16-bit LoRA performance, while FP4 lags behind NF4. On Vicuna, Guanaco 65B reaches 99.3% of ChatGPT's score, and Guanaco 33B reaches 97.8%. The table also shows the memory comparison that made the result notable: Guanaco 33B at 21GB exceeds Vicuna 13B's score while Vicuna 13B is listed as a 16-bit model using 26GB.

Training-scale evidence is concrete. The paper uses OASST1, HH-RLHF, FLAN v2, and Longform datasets, with LLaMA 7B through 65B models. For OASST1, the hyperparameter table uses batch size 16 and 1,875 steps across 7B, 13B, 33B, and 65B; for "All" mixtures, batch size rises to 64 for 65B with 2,500 steps.

Historical Effect

QLoRA changed who could adapt large open models. Before it, full finetuning a 33B or 65B model implied multi-GPU high-memory infrastructure. By moving the frozen base to 4-bit storage, keeping LoRA adapters trainable, and using paged optimizers for spikes, QLoRA made instruction tuning a single-machine task for models that previously felt cluster-bound.

Historically, this helped turn open LLM development into a broader ecosystem. Labs and individuals with one 24GB or 48GB GPU could run meaningful finetunes, compare instruction datasets, and produce chat models such as Guanaco. It also clarified a distinction that became central to efficient LLM work: training memory can be reduced dramatically without necessarily sacrificing final task performance, as long as the base representation is preserved and the trainable adapter path has enough capacity.

Limits

QLoRA mainly solves finetuning memory, not all serving or training costs. A 65B model may fit on one 48GB GPU for finetuning, but throughput, latency, and production serving economics remain separate problems. Paged optimizers rely on CPU-GPU paging; the paper reports favorable measurements for some 65B settings, but broader overhead across hardware, sequence lengths, and workloads is left to future characterization.

Quality also depends heavily on the base model, data, and benchmark. The authors note that strong Vicuna chatbot scores do not imply strong MMLU performance and vice versa. Guanaco has observed weaknesses in obscure facts, mathematics, theory-of-mind reliability, and refusal behavior. The paper performs limited responsible-AI evaluation, and its headline comparisons use GPT-4 or human judgments on specific chatbot benchmarks. The method is a major memory breakthrough, but not a guarantee of robust alignment, factuality, or high-throughput deployment.

Links