Fine-Tuning BERT and LLaMA with Hugging Face Trainer: LoRA, QLoRA, and Evaluation

Fine-tuning a 7B LLM used to require 8x A100s. With QLoRA, you can do it on a single RTX 4090 in 4 hours.

Your GPU isn't obsolete; you've just been sold the wrong playbook. The era of full-parameter fine-tuning for every task is over. Loading a 7-billion parameter model like LLaMA 2 7B in 16-bit precision eats ~14GB of VRAM just to think about training. Add gradients and an optimizer state? You're looking at 42GB+—a one-way ticket to CUDA out of memory town.

Thankfully, the paradigm has shifted. Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA (Low-Rank Adaptation) and its quantized cousin QLoRA let you train a tiny fraction of the model's weights, achieving comparable performance for a fraction of the cost. This isn't a hack; it's the new standard. Transformer architecture underpins 94% of top-performing models on 15 major benchmarks (Papers with Code 2025), and efficiently adapting these giants is now a core skill.

Let's cut through the abstract theory and get to the practical engineering: how to implement, tune, and deploy these methods using the Hugging Face Trainer without blowing up your training run.

Full Fine-Tuning vs. LoRA vs. QLoRA: Picking Your Weapon

Your choice isn't about what's "best" in a vacuum; it's about aligning method to constraint. Think of it as a trade-off triangle between performance, speed, and hardware.

Full Fine-Tuning updates every single parameter in the model. It's the brute-force approach. You need this when your target task is wildly different from the model's pre-training domain (e.g., teaching a language model to write SQL queries from scratch). It can achieve the highest possible accuracy but at a staggering cost. You'll need enough VRAM to hold the model, its gradients, and the optimizer states (like Adam, which stores two moving averages per parameter). For a 7B model in BF16, that's roughly: Model (14GB) + Gradients (14GB) + Optimizer (28GB) = 56GB. Good luck.

LoRA is the surgical strike. Instead of updating the dense weight matrices (e.g., W in y = Wx + b) in the attention and sometimes MLP layers, LoRA freezes them. It injects trainable, low-rank decomposition matrices alongside the frozen weights. During a forward pass, the output becomes y = Wx + b + (B*A)x, where A and B are the small, trainable LoRA matrices. You only train A and B, which can be 0.1% the size of the original weights. The result? You can fine-tune a 7B model on a 24GB GPU. Performance? Often within 1-2% of full fine-tuning for in-domain tasks.

QLoRA is LoRA for the rest of us. It takes the original model and quantizes it to 4-bit precision (NF4) using the bitsandbytes library. This shrinks the 7B model from 14GB to ~4GB in memory. You then apply LoRA adapters to this frozen, quantized base model. The magic trick: during training, the 4-bit weights are dequantized to a "simulated" 16-bit precision for the forward and backward passes, but the gradients are computed only for the LoRA adapters. The quantized weights are never updated. This lets you fine-tune a 7B model on a single 16GB GPU (like an RTX 4080/4090). The performance drop versus 16-bit LoRA is typically negligible.

The verdict? For 95% of instruction-following, chat, or style adaptation tasks, start with QLoRA. It's the practical default.

LoRA Theory in 5 Minutes: What r and alpha Actually Control

Forget the singular value decomposition whiteboard. You need to know two hyperparameters: r and alpha.

• r (rank): The intrinsic rank of the low-rank matrices A and B. If the original weight matrix W is of size [d x k], then A is [d x r] and B is [r x k]. A higher r means a larger, more expressive adapter. It's your model's capacity knob. Start with 8 or 16. Going to 64 rarely helps much and just makes training slower.
• alpha: This is a scaling factor. The final adapted weight is W + (alpha / r) * B*A. Think of alpha as the learning rate for the adapter. A higher alpha means the new LoRA weights are given more relative importance compared to the frozen pre-trained weights W. The rule of thumb: set alpha to 2*r as a starting point (e.g., r=8, alpha=16). This keeps the magnitude of the update stable as you change r.

In practice, r controls how many new concepts the adapter can learn, and alpha controls how loudly it speaks them relative to the original model. Tune alpha with your learning rate.

QLoRA Setup: bitsandbytes 4-bit Quantization + LoRA Config

Here's where we move from slides to code. First, install the non-negotiable tools: pip install transformers accelerate peft bitsandbytes torchmetrics.

Now, let's load a model in 4-bit and prepare it for LoRA. We'll use meta-llama/Llama-2-7b-hf as an example (ensure you have access on Hugging Face).

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
import torch


bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,  # The magic line
    bnb_4bit_quant_type="nf4",  # Normal Float 4 (recommended)
    bnb_4bit_compute_dtype=torch.bfloat16,  # Compute dtype during forward pass
    bnb_4bit_use_double_quant=True,  # Second quantization for even smaller memory
)

# Load model with quantization
model_id = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",  # Let Accelerate handle layer placement
    trust_remote_code=False,
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token  # Necessary for some models

# Define LoRA configuration
lora_config = LoraConfig(
    r=16,  # Rank
    lora_alpha=32,  # Scaling factor (alpha)
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],  # Attn layers in LLaMA
    lora_dropout=0.05,  # Dropout for LoRA layers (prevents overfitting)
    bias="none",  # Don't train bias params
    task_type="CAUSAL_LM",
)

# Wrap the model for PEFT
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # Should show ~0.1% trainable params

This script loads LLaMA 2 7B in ~4-6GB of VRAM. The target_modules are model-specific. For LLaMA, you target the query, key, value, and output projections. For BERT, you might target "query", "value".

Hugging Face Trainer: TrainingArguments That Actually Matter

The Trainer API abstracts away the training loop, but its TrainingArguments class is a minefield of defaults you must override. Here's a production-ready configuration for QLoRA.

from transformers import TrainingArguments, Trainer, DataCollatorForLanguageModeling
from datasets import load_dataset
import os

# Load a sample dataset (e.g., for instruction tuning)
dataset = load_dataset("json", data_files="your_instructions.jsonl")["train"]

# Critical Training Arguments
training_args = TrainingArguments(
    output_dir="./llama2-7b-lora-finetuned",
    per_device_train_batch_size=4,  # Limited by VRAM
    gradient_accumulation_steps=8,   # Effective batch size = 4 * 8 = 32
    warmup_steps=100,                # 10% of total steps is a good start
    num_train_epochs=3,
    learning_rate=2e-4,              # Slightly higher than full fine-tune (3e-5)
    fp16=True,                       # Use mixed precision (even with QLoRA)
    logging_steps=10,
    save_steps=500,
    evaluation_strategy="steps",     # Crucial: eval during training
    eval_steps=500,
    save_total_limit=2,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    report_to="none",                # Disable WandB/TensorBoard if you want
    gradient_checkpointing=True,     # Saves VRAM at cost of ~20% slower training
    optim="paged_adamw_8bit",       # Uses 8-bit optimizer from bitsandbytes
    lr_scheduler_type="cosine",      # Better than linear decay
)

# Use a collator for causal language modeling
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

# Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    eval_dataset=dataset.select(range(100)),  # Small eval set
    data_collator=data_collator,
    # compute_metrics=compute_metrics  # We'll add this next
)

# Start training
trainer.train()

Key Arguments Explained:

• gradient_accumulation_steps: Simulates a larger batch size by accumulating gradients over multiple steps before updating weights. Essential for stable training on small GPUs.
• optim="paged_adamw_8bit": Uses an 8-bit AdamW optimizer, saving ~4x memory on optimizer states.
• gradient_checkpointing: Trades compute for memory. It recomputes certain activations during the backward pass instead of storing them all.
• lr_scheduler_type="cosine": The 1-cycle LR schedule (a specific cosine variant) achieves 15% faster convergence, 0.8% higher final accuracy (fastai study). The Trainer's cosine decay is a good approximation.

Evaluation Callbacks: Compute Metrics Every N Steps, Not Just at End

Waiting until the end of training to discover your model is overfitting is a rookie mistake. You need metrics during training. Here's how to add a proper evaluation callback.

import numpy as np
from torchmetrics.text import Perplexity
from transformers import EvalPrediction

# Initialize metric (run on CPU to save VRAM)
perplexity_metric = Perplexity(ignore_index=tokenizer.pad_token_id).to("cpu")

def compute_metrics(eval_pred: EvalPrediction):
    """
    Compute perplexity for language modeling.
    """
    logits, labels = eval_pred
    # Shift labels and logits for next-token prediction
    shift_logits = logits[..., :-1, :].argmax(-1)  # Get predicted token IDs
    shift_labels = labels[..., 1:]

    # Calculate perplexity
    # Note: This is a simplified calculation. For production, use a proper library.
    # We'll use cross-entropy as a proxy.
    loss_fct = torch.nn.CrossEntropyLoss(ignore_index=tokenizer.pad_token_id)
    loss = loss_fct(torch.tensor(logits[..., :-1, :]).view(-1, logits.shape[-1]),
                    torch.tensor(labels[..., 1:]).view(-1))
    perplexity = torch.exp(loss).item()

    return {"perplexity": perplexity}

# Add this function to the Trainer's `compute_metrics` argument
trainer.compute_metrics = compute_metrics

Now, every eval_steps (set to 500 above), the trainer will compute and log perplexity on your validation set. A dropping perplexity is good. If training loss drops but validation perplexity rises, you're overfitting.

Real Error & Fix:

Overfitting with 98% train / 62% val accuracy Fix: Your LoRA adapters are memorizing. Add lora_dropout=0.3 to your LoraConfig. Increase dataset diversity. Apply weight decay (weight_decay=0.01 in TrainingArguments). If small dataset, use num_train_epochs=1.

Training Instabilities: Exploding Loss, Gradient Norm Spikes, Fixes

QLoRA is stable, but not immune to instability, especially with high learning rates or poor data.

• Exploding Loss in Early Training: This often manifests as loss = nan.

  • Fix: This is a classic exploding loss in Transformer training. Immediately reduce your learning rate. Start as low as 1e-5. Ensure you have gradient clipping: add max_grad_norm=0.3 to your TrainingArguments. Add a warmup: warmup_steps=100.

• Training Plateaus After Epoch 5:

  • Fix: Your learning rate has decayed to near zero. Use CosineAnnealingLR (which lr_scheduler_type="cosine" does) with a long tail. Alternatively, your dataset might have repetitive patterns causing the model to stop learning. Add a small amount of label-preserving noise to your inputs or try a different r/alpha combination.

• Vanishing Gradient in Deep Network:

  • Fix: This is less common with pre-trained Transformers but can happen if you add many new layers. Since we're using LoRA on existing layers, the pre-trained residual connections handle this. If you were modifying architecture, the fix would be: use residual connections (skip connections), He initialization for ReLU, gradient clipping norm=1.0.

Pro Tip: Always monitor gradient_norm in your logs (enable logging_steps). A sudden spike (>10.0) is a red flag for instability.

Merging LoRA Weights and Exporting for vLLM Deployment

Training is done. You have a ./llama2-7b-lora-finetuned folder containing the adapter weights (adapter_model.bin), not a full model. For efficient inference with tools like vLLM or Hugging Face's pipeline, you need a single, standard model file.

You have two options:

1. Merge and Save (Simpler Inference): This creates a new, full-precision model where the LoRA weights are added into the base weights. It's a one-time cost for faster inference later.

   from peft import PeftModel
   
   # Load the base model (in FP16/BF16, not 4-bit for merging)
   base_model = AutoModelForCausalLM.from_pretrained(
       model_id,
       torch_dtype=torch.bfloat16,
       device_map="auto",
   )
   # Load the trained PEFT model
   model = PeftModel.from_pretrained(base_model, "./llama2-7b-lora-finetuned")
   
   # Merge and save
   merged_model = model.merge_and_unload()
   merged_model.save_pretrained("./llama2-7b-merged", max_shard_size="2GB")
   tokenizer.save_pretrained("./llama2-7b-merged")
   

2. Use PEFT During Inference (Flexible): Keep the base model and adapters separate. This is lighter if you switch between multiple adapters.

   # Load for inference without merging
   from peft import PeftConfig, PeftModel
   
   config = PeftConfig.from_pretrained("./llama2-7b-lora-finetuned")
   base_model = AutoModelForCausalLM.from_pretrained(
       config.base_model_name_or_path,
       torch_dtype=torch.bfloat16,
       device_map="auto",
   )
   model = PeftModel.from_pretrained(base_model, "./llama2-7b-lora-finetuned")
   
   # Inference is the same
   inputs = tokenizer("Human: What is quantum computing?\nAssistant:", return_tensors="pt").to("cuda")
   outputs = model.generate(**inputs, max_new_tokens=100)
   print(tokenizer.decode(outputs[0], skip_special_tokens=True))
   

For vLLM deployment, you need a single model directory. Use the merge and save approach. Then, point vLLM to the ./llama2-7b-merged folder.

# Example vLLM command after merging
python -m vllm.entrypoints.openai.api_server \
    --model ./llama2-7b-merged \
    --served-model-name llama2-7b-finetuned \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9

Next Steps: From Working Script to Production Pipeline

You've gone from theory to a trained, saved model. But this is just the beginning. To move from a notebook to a robust pipeline, focus on these next steps:

1. Automated Experiment Tracking: Replace report_to="none" with report_to="wandb". Log your hyperparameters (r, alpha, lr), loss curves, and validation metrics for every run. Knowledge distillation achieves 95% of teacher model accuracy at 30% model size (average across 20 papers, 2025)—tracking helps you find such efficient configurations.
2. Systematic Hyperparameter Sweep: Use a library like optuna or ray[tune] to sweep learning_rate (1e-5 to 5e-4), r (4, 8, 16, 32), and lora_alpha (8, 16, 32). The optimal lr for QLoRA is often 10x higher than for full fine-tuning.
3. Dataset Engineering: The quality of your instruction or chat dataset is the single biggest lever on final performance. Clean your data, ensure diverse formats, and consider techniques like data augmentation for text (e.g., back-translation, synonym replacement) if your dataset is small. Remember, transfer learning reduces required training data by 10–100x vs training from scratch (DeepMind survey 2025), but you still need good data.
4. Benchmark Rigorously: Don't just eyeball the outputs. Create a small, representative evaluation set with 50-100 examples. Define clear metrics (e.g., BLEU for translation, ROUGE for summarization, exact match for QA) and run them after every training session. Compare your QLoRA model against the base model and a fully fine-tuned baseline if possible.

The barrier to entry for state-of-the-art model customization has collapsed. Your 24GB GPU is now a powerhouse. Stop waiting for cloud credits or model APIs that don't fit your use case. Load a model, configure QLoRA, and start training. The specific intelligence you need is now just a few hours and a well-written train.py script away.