What Is LoRA Fine-Tuning? Low-Rank Adaptation Explained

Fine-tuning became broadly practical when the cost of doing it fell far enough that ordinary teams could try it. LoRA is the technique most responsible for that, and it has also made per-customer model customisation economically sensible. It has not, however, changed what fine-tuning is good for, which remains the most common misunderstanding. This explainer covers both. It complements fine-tuning vs rag and what is parameter efficient fine-tuning, and reflects FISTA Solutions' approach in AI enablement work.

How does LoRA work?

The original weights are frozen. Alongside selected layers, two small matrices are trained whose product has the same shape as the weight matrix but a much lower rank, meaning far fewer parameters. At inference the adapter's contribution is added to the frozen weights.

Because only the small matrices are trained, memory and compute fall by orders of magnitude relative to full fine-tuning, and the resulting artefact is megabytes rather than gigabytes.

Property	Full fine-tuning	LoRA
Trained parameters	All	A small fraction
Memory during training	Very high	Much lower
Output artefact	Complete model	Small adapter
Serving many variants	Impractical	Practical
Quality on adaptation tasks	Baseline	Comparable
Switching cost	High	Load a different adapter

What does fine-tuning actually teach?

Form. How to respond, in what structure, with what tone, following which conventions. A model fine-tuned on well-formed examples of a specific output format will produce that format reliably, which is genuinely useful and hard to achieve with prompting alone at scale.

What it does not teach reliably is fact. Training on documents containing information does not install that information as retrievable knowledge; it shifts the model's style toward those documents while producing confident errors about their content.

When is retrieval the right answer?

Whenever the requirement is knowledge. Facts about your products, policies, pricing, customers, or documents belong in retrieval, where they can be updated when they change, cited so users can verify, and removed when they become wrong.

Fine-tuned knowledge has none of those properties. It cannot be updated without retraining, cannot be cited, and cannot be corrected except by another training run. The two techniques are complementary: fine-tune the behaviour, retrieve the facts. See fine-tuning vs rag.

How much data is needed?

Less than most teams assume, and quality dominates. Several hundred examples that consistently demonstrate the target behaviour typically outperform thousands of inconsistent ones, because inconsistency is itself learned.

Curation is therefore the main work. Examples should be reviewed for correctness, consistency of format, and absence of the patterns you do not want reproduced — including any undesirable habits present in historical data.

What is QLoRA?

LoRA applied on top of a quantized base model, which reduces memory further and makes fine-tuning feasible on smaller hardware. The quality effect is generally modest, and it has made fine-tuning of large models accessible to teams without substantial accelerator budgets.

As with any quantization, it should be evaluated on the actual task rather than assumed neutral.

How are multiple adapters served?

By keeping one base model resident and applying different adapters per request. Modern serving frameworks support this directly, and it is the property that makes per-customer or per-task customisation practical: dozens of adapters cost a fraction of what dozens of full models would.

This changes what is economically sensible. Customising behaviour per enterprise customer is a reasonable product decision with adapters and an unreasonable one with full models.

How should it be evaluated?

Against the base model with a good prompt, on the same evaluation set. The comparison teams frequently skip is whether fine-tuning beat careful prompting, and the answer is sometimes no — particularly as base models improve.

Evaluation should also check for regression on tasks outside the fine-tuning distribution, because adaptation to one behaviour can degrade others.

What are the operational costs?

Ongoing, and often underestimated. A fine-tuned adapter is tied to a base model version, so a base model upgrade means retraining and re-evaluating. Training data must be maintained. Evaluation must be rerun. Each adapter is a small artefact with a real lifecycle.

Teams should be honest about whether the gain justifies that recurring obligation, particularly where prompting achieves most of the benefit.

When is LoRA clearly the right choice?

When a consistent output format or domain-specific style is required at high volume, when prompting achieves it unreliably or only with a very long prompt, when the behaviour is stable enough not to need frequent retraining, and when the knowledge the task needs comes from retrieval rather than from training.

How does rank affect the result?

The rank of the adapter matrices controls how much capacity the adaptation has. Low ranks train fast, produce tiny adapters, and capture simpler behavioural shifts; higher ranks capture more at proportionally more cost. Most practical adaptation works well at modest ranks, and increasing rank is rarely the fix when results disappoint — the training data usually is.

How FISTA Solutions helps

FISTA Solutions separates behaviour adaptation from knowledge requirements, curates small high-quality training sets rather than large noisy ones, evaluates adapters against well-prompted base models before adopting them, and designs multi-adapter serving where per-customer customisation is warranted, through AI enablement, AI agents, and forward deployed engineers. The record behind the approach is 150+ projects for 50+ companies with 99.9% uptime.

To decide whether fine-tuning is right for your case, message FISTA on WhatsApp, or read fine-tuning vs rag.