LLM Fine-Tuning Explained: When to Fine-Tune, When to Use RAG, and When Better Prompts Are Enough

· 18 min read

Your support lead wants the AI assistant to sound like your brand instead of a generic chatbot. Your product manager has heard that you can "train ChatGPT on our docs." An engineer suggests fine-tuning an open model on two years of support tickets. A vendor is quoting a five-figure project to build a "custom LLM" for your company. Before anyone signs a contract, it helps to know what fine-tuning actually does, and what it does not.

Fine-tuning means continuing to train an existing large language model on your own examples so that its behavior changes. It is a real and useful technique. It is also one of the most misunderstood ideas in business AI. Many teams reach for it to solve problems that better prompts or document retrieval would solve faster, more cheaply, and more reliably. Others avoid it when it would genuinely help.

This guide explains fine-tuning in plain English for U.S. B2B SaaS teams: how it works, the main methods, what it is good and bad at, how it compares with prompting and retrieval-augmented generation (RAG), what it costs, the risks to plan for, and what it means for how AI engines describe your brand. If you are new to the technology, start with our practical guide to large language models and our explainer on how LLMs are trained, then come back.

Published November 19, 2026. This is an educational explainer. Provider features, supported models, and prices change often. Where this guide describes a specific method or research result, it links to the original paper or official documentation; check the current documentation before you plan a project.

What Fine-Tuning Actually Is

A large language model is built in stages. First comes pretraining, where the model learns to predict the next token across an enormous amount of text and absorbs grammar, facts, and patterns along the way. Then developers apply post-training: instruction tuning, preference training, and safety work that turn a raw text predictor into a helpful assistant. Our training guide walks through those stages.

Fine-tuning, in the business sense, is one more round of training applied after all of that, using a much smaller dataset that you supply. The model's internal numbers, called weights, are adjusted slightly so that it becomes more likely to produce outputs that look like your examples.

Three comparisons help separate fine-tuning from the other options:

A useful rule of thumb: prompting and RAG change what the model knows right now; fine-tuning changes how the model tends to behave. That distinction explains most of the successes and failures you will see.

Where Fine-Tuning Came From

Adapting a general model to a specific task is an old idea in machine learning. In 2018, the ULMFiT paper by Jeremy Howard and Sebastian Ruder showed that a language model pretrained on general text could be fine-tuned for text classification with far less labeled data than training from scratch. Google's BERT made "pretrain, then fine-tune" the standard recipe for language tasks.

The modern chat assistant is itself a product of fine-tuning. OpenAI's 2022 InstructGPT paper described fine-tuning GPT-3 on human-written demonstrations and then on human preference rankings. The authors reported that people preferred outputs from a 1.3-billion-parameter InstructGPT model over outputs from the original 175-billion-parameter GPT-3, even though it was more than 100 times smaller. That result is worth remembering: training on the right examples can matter more than raw size.

In 2023, the LIMA paper from Meta researchers and collaborators pushed the point further. They fine-tuned a 65-billion-parameter model on just 1,000 carefully selected prompts and responses. In their human study, its answers were judged equivalent to or better than GPT-4's in 43 percent of cases, and the comparison was more favorable against other assistants of the time. Their interpretation, which they called the "superficial alignment hypothesis," was that a model learns almost all of its knowledge during pretraining and fine-tuning mainly teaches it which style and format to use. That idea is debated, but it matches a lot of practical experience.

How Fine-Tuning Works, Step by Step

Most business fine-tuning projects follow the same basic path.

  1. Define the behavior. Write down exactly what should change: a tone, an output format, a classification scheme, a decision policy. "Make it better" is not a target.
  2. Collect examples. Each example is usually an input and the ideal output, often formatted as a short conversation. Examples should cover the real range of cases, including edge cases and refusals.
  3. Clean and split the data. Remove errors, duplicates, personal data you do not need, and examples that contradict each other. Hold back a separate set of examples that the model never trains on so you can measure the result fairly.
  4. Train. The model reads your examples, compares its outputs with the ideal outputs, and adjusts its weights to close the gap. This repeats over the dataset one or more times, called epochs.
  5. Evaluate. Run the held-back examples through both the original and the fine-tuned model, and compare them on the measures you defined in step one. Our guide to evaluating LLMs explains how to build a useful test set.
  6. Deploy and monitor. Route real traffic carefully, watch for regressions, and plan how you will retrain when your products or policies change.

Most of the effort is in steps one, two, three, and five. The training itself is often the shortest part.

The Main Types of Fine-Tuning

"Fine-tuning" covers several different techniques. The names matter because they solve different problems.

Supervised fine-tuning

Supervised fine-tuning (SFT) is the most common form. You provide input-output pairs, and the model learns to imitate the outputs. It is well suited to teaching a format, a voice, or a consistent way of handling a type of request. When providers or vendors say "fine-tuning" without qualification, this is usually what they mean.

Preference fine-tuning

Sometimes it is easier to say which of two answers is better than to write the perfect answer. Preference fine-tuning trains on pairs of responses labeled better and worse. The original approach, reinforcement learning from human feedback (RLHF), trained a separate reward model and then optimized the language model against it. A simpler 2023 method called Direct Preference Optimization (DPO) learns directly from the preference pairs without a separate reward model, and it is now widely available.

Reinforcement fine-tuning

For tasks where an answer can be checked automatically, such as whether code passes tests or whether an extracted value matches a known record, reinforcement fine-tuning lets the model try answers and rewards the ones that a grader marks correct. This is related to how reasoning models are trained. It is powerful when you have a reliable grader and risky when you do not, because the model learns to satisfy the grader, not necessarily your real goal.

Continued pretraining

Continued pretraining, sometimes called domain-adaptive pretraining, trains the model on large amounts of raw text from a field, such as legal filings or biomedical papers, rather than on question-and-answer pairs. It needs far more data and computing power than SFT and is mostly done by AI developers and large enterprises building specialist models.

Distillation

Distillation uses a large, expensive "teacher" model to generate example outputs, then fine-tunes a smaller, cheaper "student" model on them. The idea goes back to a 2015 paper by Geoffrey Hinton and colleagues. Businesses use it to get most of a large model's quality on a narrow task at a fraction of the running cost. Check the terms of service first: some providers restrict using their outputs to train competing models.

Full Fine-Tuning vs. Parameter-Efficient Fine-Tuning

Modern models have billions of weights. Updating all of them, called full fine-tuning, requires a lot of GPU memory and produces a complete new copy of the model for every version you train.

Parameter-efficient fine-tuning methods avoid that by freezing the original weights and training a small number of new ones. The best known is LoRA (Low-Rank Adaptation), introduced by Microsoft researchers in a 2021 paper. For GPT-3 175B, the authors reported that LoRA cut the number of trainable parameters by about 10,000 times and the GPU memory needed for training by about three times compared with full fine-tuning, while matching or exceeding its quality on the tasks they tested.

In 2023, QLoRA combined LoRA with a compressed, 4-bit version of the base model. Its authors reported fine-tuning a 65-billion-parameter model on a single 48 GB GPU, something that previously required a cluster of machines.

The practical consequences for a business are significant:

Our guide to open vs. closed LLMs explains when running and adapting an open-weight model yourself makes sense, and when a provider's hosted fine-tuning is the simpler route.

What Fine-Tuning Is Good At

Fine-tuning works best when you want a model to do something it can already do, but more consistently, more cheaply, or in a particular way. Common good fits include:

The common thread is behavior, not knowledge. If you can describe the change as "always do it this way," fine-tuning is a candidate.

What Fine-Tuning Is Bad At

The most common mistake is trying to use fine-tuning to teach a model facts, such as your current pricing, your product catalog, or this quarter's feature list.

Research is consistent on this point. In a 2023 study, Fine-Tuning or Retrieval?, Microsoft researchers compared ways of adding knowledge to models and found that retrieval consistently outperformed unsupervised fine-tuning, both for knowledge the models had seen before and for entirely new information. They also observed that models struggled to learn new facts through fine-tuning unless the same fact was presented in many different phrasings.

A 2024 study by researchers at Google and the Technion, Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, found something more concerning. Models learned fine-tuning examples containing new facts much more slowly than examples consistent with what they already knew, and as they eventually learned those new-fact examples, their tendency to make things up increased. In other words, forcing facts in through fine-tuning can make a model less reliable. Our explainer on LLM hallucinations explains why models produce confident errors in the first place.

There are practical problems too:

For facts that change or that must be traceable, retrieval is almost always the better tool.

Fine-Tuning vs. RAG vs. Prompting: A Decision Guide

Most real systems combine all three, but they answer different questions. Use this sequence as a starting point.

Start with prompting. Write clear instructions, give a few strong examples, specify the output format, and say what to do when information is missing. OpenAI's own fine-tuning documentation recommends trying prompt engineering first, because it is faster to iterate on and often enough. If a well-written prompt achieves your quality bar, stop there.

Add retrieval when the problem is knowledge. If the model gives wrong or outdated facts about your products, policies, or customers, give it the right documents at answer time. This keeps facts current, allows citations, and respects permissions. Embeddings and vector search are the usual building blocks.

Consider fine-tuning when the problem is behavior. If, after good prompting and retrieval, the model still drifts from your format, misapplies your classification scheme, or needs a long prompt on every call, fine-tuning may help. It is also worth considering when cost or speed at high volume makes a smaller specialized model attractive.

Some quick diagnostic questions:

The combination is often strongest: a fine-tuned model that reliably follows your format and voice, fed current facts through retrieval at answer time.

How Much Data You Need

Less than many people expect for behavior, and far more than they expect for knowledge.

OpenAI's documentation has stated that a fine-tuning job requires at least 10 examples and that clear improvements are typically seen from about 50 to 100 well-crafted training examples, with more examples helping further depending on the task. The LIMA results suggest that 1,000 carefully curated examples can shape a general assistant's style. For narrow classification or extraction tasks, a few hundred to a few thousand labeled examples is a common range.

Quality matters more than volume. A small set of accurate, consistent examples written or approved by your best people will usually beat a large dump of historical data. Old support tickets, for example, often contain outdated answers, agent mistakes, inconsistent formatting, and personal information. Training on them teaches the model to reproduce all of that. Curate first.

Before training, decide how you will measure success, and build an evaluation set from real cases that the model never sees during training. Without it, you cannot tell whether fine-tuning helped, hurt, or simply changed the tone.

What Fine-Tuning Costs

Costs fall into a few buckets.

Training. Hosted services from providers such as OpenAI, Google Cloud's Vertex AI, and Amazon Bedrock typically charge based on the number of tokens processed during training or on compute time. Self-hosted fine-tuning of an open model with LoRA can often run on one rented GPU for hours rather than weeks. Prices vary widely by model and provider and change frequently, so price each option against your actual dataset size.

Running the model. Fine-tuned models on hosted platforms are often priced differently from the base model, sometimes higher per token. Some platforms also charge for keeping a custom model deployed. Self-hosted models cost whatever your GPUs cost, whether or not they are busy.

People. The biggest cost is usually human time: defining the target behavior, writing and checking examples, building evaluations, and maintaining the model. Plan for a subject-matter expert's time, not only an engineer's.

Maintenance. Every time your products, policies, or the underlying base model change, you may need to retrain and re-evaluate. Providers retire older models on their own schedules, which can force a migration of your fine-tunes. Budget for this as an ongoing cost, not a one-time project.

Compare the total against the alternative. If a better prompt plus retrieval gets you 95 percent of the way for a fraction of the effort, fine-tuning may not pay back.

Risks to Plan For

Catastrophic forgetting

Training a model heavily on a narrow task can degrade its general abilities, a problem researchers call catastrophic forgetting. A model fine-tuned to write terse ticket summaries may become worse at longer explanations. Parameter-efficient methods reduce this risk but do not eliminate it. Always test the fine-tuned model on a broad sample of tasks, not only the target task.

Weakened safety behavior

A 2023 study by researchers at Princeton, Virginia Tech, IBM Research, and Stanford, Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!, found that fine-tuning GPT-3.5 Turbo on just 10 deliberately harmful examples, at a cost of under $0.20 through the public API, largely removed its safety guardrails. More importantly for ordinary businesses, they also found that fine-tuning on harmless, everyday datasets could reduce safety behavior to a lesser degree. If you fine-tune, retest refusals and safety cases, and keep your own filters in place. Our guide to LLM security and privacy covers the wider set of controls.

Data exposure

Models can memorize and repeat parts of their training data, especially rare or repeated strings such as names, email addresses, or account numbers. Remove personal and confidential data you do not need, check your provider's data processing terms, and never fine-tune on information that any user of the model should not be able to see.

Overfitting and false confidence

A model trained too long on a small dataset can memorize its examples and handle anything slightly different poorly. It may also sound more confident in your house style while being no more accurate. Measure accuracy separately from tone.

Lock-in

A fine-tune built on one provider's model usually cannot be moved to another provider. If you may switch vendors, keep your training data, evaluation sets, and prompts well organized so you can retrain elsewhere. The data, not the model, is your durable asset.

A Practical Example

Imagine a U.S. SaaS company with a 20-person support team and an AI assistant that drafts replies for agents to approve.

At first, the drafts are polite but generic, often wrong about plan limits, and inconsistently formatted. The team's first instinct is to fine-tune on 40,000 historical tickets.

A better sequence looks like this. First, the team rewrites the prompt with clear rules about tone, structure, escalation, and what to say when the answer is unknown. Formatting improves immediately. Second, it connects the assistant to the current help center and pricing pages through retrieval, so drafts quote today's plan limits with links. Most factual errors disappear, and agents can check sources with one click.

Only then does the team look at what still goes wrong. It finds that the drafts still miss the company's particular way of handling refund requests and billing disputes. A support manager writes and approves 300 model answers for those cases, and the team fine-tunes a smaller model on them. Evaluated against 100 held-back real tickets, the fine-tuned model follows policy more consistently and costs less per draft than the large general model it replaces. Facts still come from retrieval, so a pricing change next month needs no retraining.

The numbers here are illustrative, not a measured case study, but the order of operations is the lesson: prompt, then retrieve, then fine-tune what remains.

What Fine-Tuning Means for Your Brand's Visibility

Many marketing teams ask whether they can fine-tune ChatGPT, Gemini, or Claude so those systems recommend their product. The short answer is no. Fine-tuning creates a private copy of a model for your own use. It does not change the public models that your buyers use, and nothing you fine-tune reaches their answers.

What does shape public answers is the same thing that shapes the training and retrieval behind those engines: the public record. The developers' own training and post-training draw on large collections of public text, and search-connected answers retrieve current web pages at the moment a buyer asks. If you want to understand the mechanics, our guides to AI agents and tool use and RAG explain how engines fetch and use sources.

Three practical implications follow.

Your public pages are the only "training data" you control

You cannot submit a dataset to a public model. You can make sure that your own site, documentation, pricing, and trusted third-party profiles describe your product clearly, consistently, and with dates. Contradictions between pages are a common reason AI answers get facts wrong.

Other companies' fine-tunes may describe you too

Many software products embed fine-tuned versions of open models. Those models keep whatever they learned about your category at training time and are rarely updated. A clear, stable public description of what you do, for whom, and at what price reduces the chance that outdated or wrong facts persist in those systems.

Observe before and after

Because you cannot inspect public models directly, the reliable way to understand how they describe you is to observe them. Ask the same buyer-style questions on a fixed schedule, and record the engine, model, date, prompt, and exact answer, including any cited sources. That record shows whether descriptions are accurate and how they change over time. It cannot prove why an answer changed, and no method guarantees inclusion.

A 30-Day Plan for SaaS Teams

Week 1: Inventory. List every place your company uses or plans to use LLMs. For each, write down the problem in one sentence and classify it as a knowledge problem, a behavior problem, or both.

Week 2: Fix prompts and retrieval first. For knowledge problems, connect current, approved documents through retrieval. For behavior problems, rewrite prompts with explicit rules and examples. Measure each against a small set of real cases.

Week 3: Qualify fine-tuning candidates. For any remaining behavior problem, check whether you have at least 50 to 100 high-quality, consistent examples, a held-back evaluation set, a named owner, and a plan for retraining. If not, keep improving prompts.

Week 4: Run one bounded pilot and check your public record. Fine-tune one narrow task, compare it with the prompted baseline on accuracy, safety cases, cost, and speed, and decide based on the results. In parallel, ask two consumer AI engines five buyer-style questions about your category, save the dated answers, and correct any public pages that likely caused errors.

The Bottom Line

Fine-tuning changes how a model behaves. It is excellent for consistent formats, tone, classification, and making smaller models cost-effective on narrow tasks. It is a poor way to teach facts, especially facts that change, and it can increase errors and weaken safety if done carelessly. For most business problems, the right order is better prompts first, retrieval for knowledge, and fine-tuning only for the behavior gaps that remain, always measured against a held-back test set.

For brands, the key point is simpler: you cannot fine-tune the public AI engines your buyers use. What they say about you depends on the public record they learn from and retrieve. If you want to see what AI engines currently say about your brand, and which sources they rely on, book a strategy call with LLM Recommend.

Frequently Asked Questions

What is fine-tuning an LLM?

Fine-tuning means continuing to train an existing large language model on your own examples so its behavior changes. The model's weights are adjusted so it becomes more likely to produce outputs like your examples, such as a fixed format, a house tone, or a consistent classification scheme.

Is fine-tuning better than RAG?

They solve different problems. Retrieval-augmented generation gives a model current documents at answer time, so it is better for facts that change, citations, and permissions. Fine-tuning changes behavior, such as format and tone. Research has found retrieval consistently outperforms fine-tuning for adding knowledge, and many strong systems combine both.

Can I fine-tune ChatGPT on my company's documents?

You can fine-tune some OpenAI models through its API to create a private model for your own use, but it does not change the public ChatGPT that other people use. For answering questions from your documents, retrieval is usually more accurate and easier to keep current than fine-tuning.

How much data do I need to fine-tune an LLM?

For behavior changes, less than many expect. OpenAI's documentation has stated that a job needs at least 10 examples and that clear improvements typically appear with about 50 to 100 well-crafted examples. Narrow classification or extraction tasks often use hundreds to thousands. Quality and consistency matter more than volume.

What are the risks of fine-tuning?

The main risks are higher hallucination when forcing in new facts, loss of general ability, weakened safety behavior, memorized personal or confidential data, overfitting to a small dataset, and lock-in to one provider's model. Test on held-back cases and safety prompts before deployment.

Can fine-tuning make AI engines recommend my brand?

No. Fine-tuning creates a private model and does not change the public AI engines buyers use. Those engines draw on public text and retrieved web pages, so the practical lever is a clear, consistent, dated public record about your product, plus regular observation of what engines say. No method guarantees inclusion.