Fine-Tuning an LLM on Your Own Data, Locally: What It Actually Costs in 2026

There is a moment in every serious private-AI project where retrieval stops being enough. The model answers from your documents but doesn't sound like your organization, doesn't follow your output format, doesn't apply your domain's judgment. The next step — fine-tuning an LLM on your own data — has a reputation for requiring cluster budgets and ML-team headcount. In 2026, that reputation is roughly three years out of date.

This post is the narrative companion to Guide 50 and the ten nodes we added to the index this week. The short version: a 7B model fine-tunes in about 5 GB of VRAM, the tooling has consolidated into four serious open-source options, and the two things most likely to hurt you — catastrophic forgetting and teacher-model licensing — are both manageable if you know they exist. Everything below is sourced; where a number is a vendor's own claim, we say so.

Fine-tuning vs RAG: the decision people get backwards

The most-asked question in this space — fine-tuning vs RAG, when to use what — has a structural answer, not a fashionable one.

Retrieval keeps knowledge outside the model: your documents sit in storage you control, get embedded, and enter the context window at question time. Nothing about the model changes, nothing needs a GPU beyond inference, and updating knowledge means updating an index, not retraining. That is the right default for facts — especially facts that change.

Fine-tuning changes the model itself. That is the right tool for behavior: output format, domain vocabulary, judgment patterns, style, tool-use conventions — the things you cannot reliably prompt or retrieve your way into. The catch is that changing the model has a measurable price. Luo et al. documented that fine-tuning degrades capabilities outside the target domain — knowledge, reasoning, and reading comprehension all drift — and that in their 1B–7B study range the effect got worse as models got larger. The model you sharpen on your data gets duller elsewhere. That's not a bug in your training run; it's catastrophic forgetting, and it's the oldest known failure mode of neural-network training.

So the honest decision rule: retrieve for knowledge, fine-tune for behavior, and when you fine-tune, do it in a way that bounds the forgetting. Which brings us to the method that dominates the local stack.

LoRA and QLoRA: why "on hardware I own" became realistic

Full fine-tuning updates every weight in the model. LoRA (Hu et al., 2021) freezes the pretrained weights and trains small rank-decomposition matrices injected into each layer — up to 10,000× fewer trainable parameters on GPT-3-scale models, with no added inference latency once the adapter is merged back in. QLoRA (Dettmers et al., 2023) pushed the frozen base model down to 4-bit NormalFloat and fine-tuned a 65B model on a single 48 GB GPU while matching 16-bit quality.

The 2024 result that should shape your expectations: LoRA learns less and forgets less (Biderman et al., TMLR). At common ranks, adapters underperform full fine-tuning on the target domain — while preserving far more of the base model's general capability. For most private fine-tunes, that trade is exactly the one you want to make: "very good on my domain, still competent everywhere else."

QLoRA VRAM requirements, 7B to 70B

The numbers that answer "what GPU do I need to fine-tune an LLM," per Unsloth's requirements table:

Model size QLoRA (4-bit) LoRA (16-bit)
7B ~5 GB ~19 GB
14B ~8.5 GB —
32B ~26 GB —
70B ~41 GB ~164 GB

Read that table as a hardware-shopping map. A single consumer GPU covers 7B–14B in 4-bit. One workstation card covers 32B. A single 48 GB card reaches 70B. LLaMA-Factory's published table agrees on the shape: a 7B model needs roughly 60 GB for full fine-tuning, 16 GB for 16-bit LoRA, 6 GB for 4-bit QLoRA. The gap between "full" and "QLoRA" is the gap between a cluster line-item and a gaming PC.

Unsloth vs Axolotl vs LLaMA-Factory (vs TRL): pick by hardware shape

The framework comparison people search for has a clean resolution once you stop comparing feature lists and start comparing hardware shapes:

  • One consumer GPU → Unsloth. Custom Triton kernels, the VRAM floor in the table above, 500+ supported models, GRPO and full fine-tuning alongside LoRA/QLoRA. Its speed and memory numbers are vendor-published claims, but the requirements table has held up in community use. Know the licensing split: the core library is Apache-2.0; the Studio UI is AGPL-3.0. Multi-GPU is possible but unofficial.
  • Multi-GPU, repeatable pipelines → Axolotl. YAML-configured runs (v0.18.0, Apache-2.0), FSDP1/2 and DeepSpeed, the broadest RL/preference-method coverage of the three, and — uniquely in this group — experimental native dataset loading straight from S3/GCS/Azure. If your training corpus already lives in an object-storage data lake, Axolotl is the one that meets it there.
  • Broadest model coverage plus a WebUI → LLaMA-Factory. v0.9.5, Apache-2.0, LoRA/QLoRA/DoRA plus DPO/PPO/KTO/ORPO/SimPO, and dataset_info.json can pull training files from S3/GCS. The WebUI makes it the lowest-friction entry point for a first fine-tune.
  • Library-level control → Hugging Face TRL. The SFT/DPO/GRPO/KTO trainer layer (v1.9.2, Apache-2.0) that the frameworks above build on. Choose it when you're composing a training pipeline in Python rather than driving one from a config file.

One warning the ecosystem has been slow to absorb: torchtune is not a foundation to build on. PyTorch wound down its development in mid-2025 (final release v0.6.1). It remains a readable reference implementation; new projects should start elsewhere.

Aligning on your own preference data: DPO killed the RLHF excuse

Instruction-tuning teaches the model your tasks. Making it prefer your organization's idea of a good answer used to require RLHF — a separate reward model, a PPO loop, real infrastructure. Direct Preference Optimization (Rafailov et al., 2023) collapsed that into supervised training on prompt / chosen / rejected triples. Your preference data — the pairs that encode what "good" means in your domain — never leaves your network, and TRL ships the trainer with fifteen documented loss variants. GRPO, the DeepSeekMath-lineage method that trained DeepSeek-R1, is the reasoning-flavored sibling for when you can score outputs programmatically instead of labeling pairs.

Synthetic training data: check the teacher's license before you distill

Most private fine-tunes don't have 50,000 hand-written examples. The standard move is synthetic training data generation: a strong teacher model reads your documents and generates instruction/response pairs in the Self-Instruct shape; you curate and train on the output. Tooling is mature — distilabel (Apache-2.0) and NVIDIA NeMo Curator cover generation and curation.

The part people skip is the license check, and the differences are material:

  • DeepSeek-R1 is MIT-licensed and explicitly permits distillation — the model card says so in plain language.
  • Qwen3 is Apache-2.0: outputs are yours.
  • Llama 4's community license attaches naming and attribution conditions to derivatives.
  • Gemma's terms treat models trained on its synthetic output as Model Derivatives, subject to Gemma's restrictions.
  • The Gemini API prohibits using outputs to train competing models — API teachers and open-weight teachers are not interchangeable, legally.

If autonomy is the point of the exercise, the teacher's license is part of the architecture, not paperwork. An MIT or Apache-2.0 teacher keeps the resulting model unambiguously yours.

The storage side: shards, checkpoints, and adapters

A fine-tuning loop is also a storage workload, which is why this index covers it. Three patterns matter:

  1. Corpus that fits on NVMe → simple files. JSONL or Parquet, no ceremony.
  2. Streaming from object storage → shard formats. WebDataset's tar-shard convention streams training samples straight from S3 via pipe: URLs; MosaicML Streaming's MDS format is the main alternative. Sequential shard reads instead of per-sample requests is the difference between feeding the GPU and starving it — the same data-loading physics that governs pre-training governs your fine-tune.
  3. Artifacts → version everything. A LoRA adapter is megabytes against a base model's gigabytes. Keep per-task adapters in a checkpoint/artifact lake, branch datasets with lakeFS-style versioning, and serve many adapters off one base model — vLLM hot-loads adapters at runtime behind an OpenAI-compatible endpoint.

Managing the forgetting

You cannot fully avoid catastrophic forgetting; you can bound it and measure it. The current mitigation stack, in rough order of adoption cost: prefer adapters over full fine-tuning (the Biderman result above); mix general data into your training set — replay-based methods like FOREVER (ACL 2026) formalize how much and which; and merge rather than overwrite when combining capabilities (BaM). Specialized approaches like SSU reported cutting degradation from 20.3% to 3.4% at 7B scale in their setting. Whatever you adopt: benchmark the base model on a general suite before training, and re-run it after. The delta is your forgetting bill, and you want to see it before your users do.

The point

Fine-tuning an LLM on your own data, locally, is no longer the expensive path — it's the sovereign one. The weights are yours, the data never left, the adapter is a file you can version, and the whole loop — corpus, shards, checkpoints, adapters — runs on storage and hardware you control. The index now maps every piece: start at Guide 50, and follow the edges.

Sources