Fine-tuning open-source LLMs like Llama 3 enables them to match or exceed proprietary models for domain-specific tasks, custom formats, and tone styling.
To do this on consumer hardware (e.g., a single RTX 390/4090), we use **QLoRA (Quantized Low-Rank Adaptation)**.
How QLoRA Works
QLoRA freezes the base model parameters in 4-bit precision and inserts tiny trainable Low-Rank Adaptation (LoRA) weight matrices in the attention modules.
┌──────────────┐
│ Base Weights │ (Frozen 4-bit NormalFloat)
└──────┬───────┘
│
[Input Activation]
┌──────┴───────┐
▼ ▼
[Base Output] [LoRA Path (Trainable 16-bit)]
│ ┌───────┐
│ │ A │ (Low-Rank down-projection)
│ └───┬───┘
│ ▼
│ ┌───────┐
│ │ B │ (Low-Rank up-projection)
│ └───┬───┘
▼ ▼
[Final Output Activation]---
Step 1: Python Dependencies Setup
Create a virtual environment and install the required PyTorch, PEFT, and Hugging Face libraries:
pip install torch torchvision torchaudio
pip install transformers peft bitsandbytes datasets trl accelerate---
Step 2: Loading Base Model in 4-bit
We configure `BitsAndBytesConfigs` to load Llama 3 in 4-bit precision with double quantization:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
model_id = "meta-llama/Meta-Llama-3-8B"
# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)
# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token---
Step 3: LoRA Hyperparameters
Define the Adapter configuration targeting attention module layers (e.g. `q_proj`, `v_proj`):
from peft import LoraConfig, get_peft_model
lora_config = LoraConfig(
r=16, # Rank dimension
lora_alpha=32, # Scaling parameter
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
# Wrap base model with PEFT adapters
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: ~17M | all params: ~8B---
Step 4: Launching SFTTrainer
Format your dataset into instructions and kick off Supervised Fine-Tuning: