In This Handbook

Fine-tuning LLMs

Previously, we discussed training in general. Now let’s focus on fine-tuning; the practice of taking an already pretrained LLM and training it further on a specific dataset to adapt it to certain tasks or domains. Fine-tuning can significantly improve performance for specialized applications, but it’s not always the right choice. In this lesson, we’ll cover when to fine-tune vs. when not to, different types of fine-tuning (from full model fine-tuning to parameter-efficient methods), and the trade-offs involved. We’ll also consider real-world constraints like the cost of fine-tuning and deploying multiple models, as well as governance issues (like maintaining safety and compliance after fine-tuning). Finally, we’ll compare fine-tuning with other methods like prompting and retrieval one more time, framing it as a “triangle” of options to achieve desired results. This will help you decide the best approach for a given use case.

ⓘ Fine-tuning shifts probabilities, not rules

Fine-tuning does not add explicit rules or facts to a model. It adjusts the model’s internal probability distributions so that certain outputs become more likely across many contexts. This means its effects are global and implicit, not localized or easily constrained.

When should you fine-tune an LLM?

Fine-tuning is powerful when you have narrow, well-defined tasks or domains where you need the model to consistently perform at a high level. Some scenarios where fine-tuning shines:

Domain specialization: To make an LLM an expert in a specific field (e.g., legal, medical, finance), fine-tuning it on domain-specific data (like legal summaries or medical Q&A) significantly improves accuracy and jargon awareness. While the base model has general knowledge, the fine-tuned version gains the necessary nuance and style for clinically relevant or specialized answers.
Task specialization: Fine-tuning specializes a base model for a specific field, like medicine, leading to more relevant answers. For tasks such as sentiment analysis or named entity recognition, a fine-tuned model trained on labeled examples consistently outperforms prompting a generic model, essentially creating a bespoke model for that task.
Style or tone consistency: Fine-tuning creates a custom model for a specific task, often outperforming generic models. For example, fine-tuning a model on labeled reviews can directly output “positive” or “negative” sentiment instead of relying on a prompt. It also guarantees consistent style or tone, crucial for a company chatbot to adopt a specific brand voice and adhere to guidelines by training it on style-reflective examples.
Improving weak areas: Fine-tuning ensures consistency for a specific persona (e.g., friendly customer service or a formal bank assistant), instead of prompting for style every time. It’s also useful for improving weak areas if the base model struggles with certain inputs (like a language or long documents) and you can obtain more training data.
When you need repeatable, consistent outputs for the same inputs: Fine-tuning ensures repeatable, consistent outputs for the same inputs, reducing the variability sometimes seen in generative models, especially for repeated, formatted queries.

However, it’s important to emphasize: fine-tuning is best when you have a clear idea of the desired output behavior and sufficient data to demonstrate that behavior. It’s also typically used when you have hundreds or thousands of examples to learn from. If you have only 10 examples, you probably shouldn’t fine-tune as the model might just overfit those 10 and not generalize.

ⓘ Irreversibility matters

Unlike prompting or retrieval, fine-tuning is not easily reversible. Once a model is fine-tuned, undoing its effects usually requires retraining from an earlier checkpoint. This makes fine-tuning a higher-risk, higher-commitment decision.

When not to fine-tune

There are many cases where fine-tuning is not the ideal approach:

Adding factual knowledge (small scale): For a few new facts (like a CEO change or discovery), fine-tuning the whole model is an expensive, often detrimental overkill. Instead, supply new facts at runtime (via prompt or retrieval). Fine-tuning is for deeply embedding permanent knowledge in the model’s “memory.” For numerous or changing facts, use a database instead of constant fine-tuning.
Small data or rapid iteration needs: Fine-tuning needs ample data; small datasets lead to overfitting, where few-shot prompting might be better as the model retains general knowledge. Also, for frequently changing requirements, prompt engineering or system rules are quicker to adjust than constantly retraining a fine-tuned model. You might iterate with prompts initially, then fine-tune once requirements stabilize and enough data is gathered.
Tasks where prompting already works well: If an off-the-shelf model works well with prompt engineering (e.g., simple summarization), fine-tuning may be unnecessary. The quality improvement from fine-tuning might not justify the effort and cost. Note that some providers charge differently for fine-tuning and using fine-tuned models.
When you value the model’s versatility: Heavy fine-tuning makes a model more specialized but often less flexible, potentially degrading performance on unrelated tasks or reducing creative output by narrowing its distribution. For generalist assistants handling varied tasks, relying on prompting or retrieval with the base general model is often better than aggressive niche fine-tuning.
Resource or deployment constraints: Fine-tuning large models (e.g., tens of billions of parameters) is often expensive, time-consuming, and creates a new model version, complicating deployment and infrastructure. Avoiding fine-tuning simplifies things. For example, OpenAI’s GPT-4 is powerful enough out-of-the-box and can be guided via prompts for many tasks, often making fine-tuning unnecessary (and it is not publicly available as of writing)..
Privacy concerns: Fine-tuning models with sensitive data (e.g., user logs, proprietary documents) raises privacy concerns, especially with third-party APIs. Retrieval methods may be safer, keeping data local and only using relevant parts during queries. Additionally, fine-tuning on user data risks the model acquiring user biases or style issues.

So a good rule of thumb: Don’t fine-tune just because you can. Try prompts and other methods first. Fine-tune when you’ve identified a clear gap that fine-tuning can fill and you have enough high-quality data to do it.

Types of fine-tuning

ⓘ Always measure before and after

Fine-tuning should always be evaluated against a baseline, such as the base model with prompting. Improvements are task-specific, and fine-tuning can introduce regressions in areas you did not intend to change.

Not all fine-tuning is equal. There are a few ways to do it:

Supervised fine-tuning (full model): Full-model fine-tuning involves retraining a pretrained model on a labeled dataset for a specific task, updating all weights. For instance, you could fine-tune a model on 50k labeled customer support emails. This method is computationally and memory-intensive, especially for large models, and requires careful setting to avoid overfitting. However, it offers the best performance if sufficient data is available, as all layers adapt to the new task.
Instruction fine-tuning: Instruction fine-tuning can customize a model to adhere to a specific instruction format using instruction-response pairs. This is common with open-source models (e.g., LLaMA vs LLaMA-Instruct). For a custom assistant, gathering and fine-tuning on Q&A or conversation data improves its ability to follow instructions conversationally. This is a form of supervised fine-tuning focused on dialogue/prompt-following tasks.
Parameter-efficient fine-tuning (PEFT): To cut costs, parameter-efficient fine-tuning methods update only a small portion of a large model’s parameters. Techniques include:
- LoRA (Low-Rank Adapters): It inserts and trains small matrices (adapters) while freezing the original weights.
- Prefix-tuning/prompt-tuning: It learns virtual tokens/vectors to condition the model. These methods allow fine-tuning on consumer GPUs, require less memory and time, and enable multiple specialized “heads” per model copy. While performance might be slightly lower.
- Full fine-tuning: The difference is often minimal, and they require managing the extra adapter components during inference.

Reward model fine-tuning / RLHF: RLHF (Reinforcement Learning with Human Feedback) adapts a model using reinforcement learning and a reward model fine-tuned from human preferences. Primarily for alignment, it’s a post-training technique not typically done by users due to the extensive human feedback data required.

ⓘ Garbage in, permanent bias out

Fine-tuning amplifies patterns in the data it is trained on. Inconsistent labels, biased examples, or poorly written outputs can permanently degrade model behavior. High-quality, representative data is more important than sheer volume.

One can also combine approaches: e.g., do a LoRA fine-tune for each specific task you have, and possibly keep the base model unchanged and just swap LoRA modules (this is like a “plug-in” approach to specialization). This is quite useful in practice, as it avoids making a fork of the whole model for each use case. For example, you might have a base model and one LoRA for legal documents, one LoRA for casual chatbot, etc., and enable the one you need.

Trade-offs: Fine-tuning vs. prompting vs. retrieval

Let’s revisit and expand the triangle analogy, i.e., three ways to adapt an AI to a task: Prompt Engineering, fine-tuning, and retrieval-augmented generation (RAG).

Note that we will discuss RAG in detail in upcoming lessons.
Picture a triangle where each corner is one of those techniques. Each has strengths and weaknesses:

The three main Generative AI approaches are:

Prompting (no model change): Fast iteration, leverages existing model knowledge, but can hit performance limits, use up context space with long prompts, and result in variability. Best for quick use of existing capabilities and when no training data is available.
Fine-Tuning (change model): High upfront cost and data requirement, but yields a specialized, high-performing model for a specific task. Usage is simpler post-training. Downsides include training cost, overfitting risk, and maintenance. Ideal for a large, stable, specialized task (e.g., dedicated customer support chatbot).
Retrieval (RAG – external knowledge): Uses a knowledge base (like a vector database) to provide the model with external, up-to-date, or proprietary information at query time, without baking it into the model’s parameters. Great for factual tasks where the source of truth is critical (e.g., company documentation Q&A). It adds complexity in maintaining the database and retrieval pipeline. Superior to fine-tuning for new or changing facts.

ⓘ Prompts behave differently after fine-tuning

Because fine-tuning changes the model’s internal distributions, prompts that worked well before fine-tuning may behave differently afterward. Prompting and fine-tuning interact, and both often need adjustment together.

These are not mutually exclusive. Many solutions use all three: e.g., you fine-tune a model to have a good conversational style and understanding of your domain’s general knowledge, you still use retrieval to give it current facts or extensive documents to refer to, and you prompt it with specific instructions for each query. For example, a legal AI assistant might be a fine-tuned model on legal text, but when asked about a particular case, it retrieves the text of that case and includes it in the prompt, and the prompt says “Answer with reference to the provided case law.”

A concrete trade-off scenario:

Without fine-tuning, you find you have to put a long prompt like “You are a helpful assistant for legal advice. Always cite at least one law or case. Use a formal tone…” etc., on each query. The answers are okay, but sometimes tone or format is off.
Fine-tuning establishes a consistent style once, saving tokens compared to repeated lengthy prompts, and ensures reliable formatting. Retrieval handles fresh information, like new laws, by updating the database instead of requiring a new fine-tuning run. Fine-tuning provides general style/knowledge up to a point; retrieval ensures freshness; prompting clarifies the user’s query or adds minor instructions.

Real-World constraints and considerations:

Fine-tuning Large Language Models (LLMs) has several considerations:

Cost: Significant. Fine-tuning needs specialized hardware (GPUs, VRAM) and can incur high cloud service charges ($X/1k tokens). Serving the fine-tuned model and integration also add engineering costs. Cost scales with model and data size.
Inference speed: Fine-tuning can actually reduce latency. It allows for shorter prompts (e.g., 50 tokens vs. 500 tokens of examples), meaning less work per inference. Furthermore, a smaller fine-tuned model (e.g., 7B) may replace a much larger general model (e.g., 70B) for better speed and cost at run-time.
Governance and safety: Fine-tuning risks degrading safety mechanisms or alignment. Unfiltered training data (e.g., containing foul language or political bias) can make the model less compliant. It’s essential to include safe behavior examples in training data. Providers often have guardrails, but self-hosting requires rigorous monitoring and testing for issues like hallucination or instruction following fidelity.

Multiple vs. unified models: Organizations must choose between a single fine-tuned model for all tasks (easier to manage, but risks domain mixing) and separate models/adapters for each task (better optimization, but more overhead). Parameter-efficient fine-tuning (like LoRA) offers a balance by keeping one base model with swappable, small weight add-ons.

ⓘ Fine-tuning is a second or third step

In practice, teams often start with prompting, then add retrieval, and only fine-tune once simpler approaches fail to meet requirements. Fine-tuning is powerful, but it is rarely the first tool to reach for.

Fine-tuning is a valuable tool, but use it strategically. Choose it for clear benefits like significant accuracy gains (e.g., 85% to 95%), major prompt simplification, or firm use-case tailoring. Always compare against simpler alternatives (like clever prompting or retrieval steps) which might offer sufficient benefits (e.g., 80%) with less effort.

We can frame the prompt vs. tuning vs. retrieval triangle visually. Imagine each corner labeled:

Prompting: Quick, no training, broad use, but can be less accurate and rely on model’s existing knowledge.
Fine-tuning: Precise, high performance in domain, but requires data & training, less flexible if requirements change.
Retrieval (Knowledge augmentation): Access to up-to-date info and detail, requires infrastructure and reliable data source, model must integrate info correctly.

Real-world solutions often blend approaches, combining prompting, retrieval, and fine-tuning. Examples include retrieval + prompting (like Bing Chat), fine-tuning + retrieval (internal chatbot), or fine-tuning + prompting (instruction-following model).

Crucially, always evaluate fine-tuned models thoroughly. Compare the fine-tuned model’s test set performance against the base model’s prompting results to ensure the improvement is significant and worth the effort, avoiding over-engineering.

Curious about how to fine-tune an LLM from scratch? Look no further! Dive into: Fine-Tuning LLMs Using LoRA and QLoRA

After five lessons, we now have a comprehensive understanding of generative AI and LLMs: what they are, how they work, and how to use them effectively. We covered the big idea (Lesson 1), LLM inner workings (Lesson 2), practical prompting (Lesson 3), training processes (Lesson 4), and model customization (Lesson 5). This foundation should provide a correct mental model of Generative AI: it’s a powerful, data- and algorithm-driven predictive engine that requires careful guidance (via prompts, training, and system design) to produce reliable, valuable applications

Previous Lesson Next Lesson

Fahim Ul Haq

I’m Fahim, a software engineer specializing in generative AI, system design, and responsible AI adoption.

View Profile

Agentic AI