Fine-tuning Llama 3 with Unsloth: local, cheap, actually works

Unsloth is the tightest fine-tuning stack in 2026 for people who don't want to think about GPU orchestration. It runs on a single consumer GPU (a 3090 fine-tunes Llama-3.1-8B; an H100 handles 70B), it's ~2x faster than vanilla PEFT, and the API is clean.

Prereqs

• A GPU with 24GB+ VRAM
• A dataset in JSONL: `{"instruction": "...", "response": "..."}`
• ~1-4 hours depending on dataset size and model

The full flow

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments

model, tokenizer = FastLanguageModel.from_pretrained(
  model_name = "unsloth/llama-3.1-8b-bnb-4bit",
  max_seq_length = 2048,
  load_in_4bit = True,
)
model = FastLanguageModel.get_peft_model(
  model,
  r = 16,
  lora_alpha = 32,
  target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)

trainer = SFTTrainer(
  model = model,
  train_dataset = ds,
  dataset_text_field = "text",
  max_seq_length = 2048,
  args = TrainingArguments(
    per_device_train_batch_size = 2,
    gradient_accumulation_steps = 4,
    warmup_steps = 5,
    num_train_epochs = 3,
    learning_rate = 2e-4,
    fp16 = True,
    logging_steps = 10,
    output_dir = "./out",
  ),
)
trainer.train()

# Save adapter
model.save_pretrained("my-adapter")
tokenizer.save_pretrained("my-adapter")

Deploying

Two options: 1. Together AI dedicated endpoint — upload the adapter, get an OpenAI-compatible URL. 2. Self-host with vLLM — vllm serve unsloth/llama-3.1-8b --enable-lora --lora-modules mine=./my-adapter

Together is easier; self-host is cheaper past ~1M tokens/day.

Common pitfalls

• Overfitting: 3 epochs is usually enough for domain adaptation. More and you erode general capability.
• Bad dataset formatting: the instruction/response text must include EOS tokens or the model won't stop generating.
• Wrong LoRA rank: 16 is the sweet spot for most tasks. 8 for narrow style/tone tuning, 64+ only for major capability shifts.