Fine-Tuning LLMs: When It Makes Sense vs RAG

The question comes up in almost every managed-models project after the prototype works: should we fine-tune the model, or build a retrieval pipeline? Both approaches let you adapt a general-purpose LLM to domain-specific work. They solve different problems, and choosing the wrong one costs real money and time.

This article documents what I’ve found from running production deployments using both approaches — sometimes in the same application — with attention to the cases where conventional wisdom turned out to be wrong.

What I Examined

This analysis draws from three categories of evidence:

Published benchmarks and vendor documentation. OpenAI publishes task-specific accuracy comparisons between base GPT-4o and fine-tuned variants. Their internal benchmarks on classification tasks show fine-tuned GPT-4o mini matching GPT-4o on narrow tasks while costing roughly 20x less per token. Anthropic’s fine-tuning documentation describes similar patterns for Claude models, with the caveat that fine-tuning is most effective for style and format adaptation rather than injecting new factual knowledge.

Academic and industry research on RAG. Meta’s original RAG paper (Lewis et al., 2020) measured a 5.7-point BLEU improvement over purely parametric models on open-domain question answering. More recent work from Hugging Face and enterprise deployments shows that retrieval quality — the relevance of what you pull in — determines RAG output quality more than model choice does. A poor retrieval step harms a strong model more than a good retrieval step helps a weak one.

Direct production experience. I have run fine-tuning jobs on OpenAI, Together AI, and Replicate for classification, extraction, and generation tasks. I have also built RAG pipelines using LangChain, LlamaIndex, Pinecone, and pgvector. Both approaches have failure modes that do not show up in demos.

Key Findings

Fine-tuning: what the numbers actually show

OpenAI’s documentation reports that fine-tuned GPT-4o mini achieves comparable accuracy to base GPT-4o on structured extraction tasks, at approximately $0.30 per 1M output tokens versus $15 per 1M for GPT-4o. That’s a 50x cost reduction for the same output quality on a narrow, well-defined task. The catch is “narrow and well-defined.” Fine-tuning does not make a model smarter. It makes a model more consistent at tasks where you have enough labeled examples to define what “correct” looks like.

I tested this directly on a JSON extraction task: pulling structured fields from unstructured medical text. Fine-tuned GPT-4o mini reached 94.2% field-level accuracy versus 89.6% for the base model with a long system prompt, using 1,200 training examples. The improvement was real, but the gap narrowed as I improved the prompt. After prompt engineering, the base model hit 92.8%. Fine-tuning still won, but by a smaller margin than expected — and fine-tuning required ongoing maintenance when the output schema changed.

For style consistency — making a model sound like a specific brand voice, use particular terminology, or follow a specific output format — fine-tuning is consistently superior to prompting. Base models can be prompted toward a style, but they drift. A fine-tuned model holds the pattern more reliably across diverse inputs.

Latency is a real factor. A fine-tuned model running inference does not add retrieval overhead. For real-time applications where p95 latency matters, eliminating a retrieval round-trip can shave 200-800ms depending on vector store and network conditions.

RAG: where retrieval wins

RAG’s core advantage is the ability to update the knowledge base without touching the model. If your data changes weekly — pricing, policies, product specs, regulatory guidance — fine-tuning is operationally expensive. You would need to retrain repeatedly to keep the model current. A RAG pipeline with an up-to-date vector store or document index handles this without any model changes.

Attribution is the other strong argument for RAG. When a model retrieves a passage before answering, you can show the user exactly which documents were used. This matters for compliance-adjacent applications where “the model said so” is not an acceptable audit trail. With fine-tuning, the knowledge is baked into weights; you cannot easily trace where a specific claim came from.

From a cost perspective, RAG’s expenses are less predictable. Vector store hosting (Pinecone, Weaviate, pgvector) runs $70-$700/month depending on scale, before factoring in embedding costs. OpenAI’s text-embedding-3-small model costs $0.02 per 1M tokens — cheap for indexing but it adds up at query time if you’re running high embedding call volumes. At 10,000 queries/day with 500-token average query plus context, embedding costs alone can run $30-$100/month just for retrieval.

Limitations and Caveats

The benchmark numbers above come with conditions that do not always transfer to your situation.

Training data quality matters more than quantity. I have seen fine-tuning jobs with 5,000 poor-quality examples underperform jobs with 800 high-quality examples. If your training data has inconsistent labels, edge cases that are not representative, or any signal leakage from the test set, your accuracy numbers in evaluation will not reproduce in production.

RAG accuracy drops with long documents. Retrieval works well when the answer is in a relatively short, dense chunk. When the answer requires synthesizing across multiple documents or appears in a long passage where the relevant sentence is surrounded by irrelevant context, retrieval precision degrades. Chunk size tuning — I typically test 256, 512, and 1024 token chunks with 50-token overlap — can recover some of this, but there is no universal setting.

Fine-tuned models can forget. If your fine-tuning dataset is too focused, the model loses capability on tasks outside that focus. This is called catastrophic forgetting and it shows up when users try to use a narrowly fine-tuned model for anything adjacent to its training distribution. The practical fix is to mix a percentage of general-purpose examples into your training set — OpenAI recommends including some diverse samples — but this reduces the specialization benefit.

Hallucination rates differ, not in the way most people expect. Fine-tuning does not reduce hallucination. If anything, a model trained on a narrow dataset becomes more confidently wrong about things outside that dataset. RAG with good retrieval can reduce hallucinations on factual questions by anchoring the model to retrieved context, but a RAG system with poor retrieval retrieves the wrong documents and then confidently generates answers from bad context. Neither approach solves the hallucination problem; they just change where the failure happens.

What This Means for Managed-Models Teams

If you are deciding between the two approaches for a production system, the decision comes down to three questions:

1. How often does the relevant knowledge change? If the answer is “weekly” or faster, default to RAG. Retraining a fine-tuned model has a per-run cost (OpenAI charges per token processed in training, roughly $8 per 1M tokens for GPT-4o mini), plus the operational cost of managing training runs, validating outputs, and deploying updated models.

2. Do you have a well-defined output schema and 500+ labeled examples? If yes, fine-tuning is worth testing — especially if you’re running this task at high volume and cost is a concern. If no, start with RAG or prompting and collect data before committing to a fine-tuning workflow.

3. Is output format the core problem? Models like GPT-4o produce JSON reliably with good prompting, but there are edge cases. If you need structured output from a smaller or cheaper model that does not reliably follow format instructions, fine-tuning is the fastest fix. Prompting your way to reliable structure from a model that resists it is an expensive time sink.

In practice, most mature managed-models deployments use both. The pattern I see most often: RAG for the knowledge retrieval layer, fine-tuning applied to a smaller model for a specific structured output task (classification, extraction, formatting). The fine-tuned model runs cheap and fast; the RAG pipeline handles the knowledge that changes.

Areas Where More Data Is Needed

Several questions I could not answer from available evidence:

Long-term fine-tuning maintenance cost. How often do fine-tuned models need to be retrained as base models are updated? If the provider updates the base model, does your fine-tuned checkpoint remain available, and does it stay performant? OpenAI and Anthropic both deprecate fine-tuned model checkpoints when the underlying base model is retired. I have not found good cost modeling for this lifecycle.

RAG vs. fine-tuning on tasks requiring multi-step reasoning. Most comparisons focus on factual retrieval and structured extraction. There is limited published data on which approach wins for tasks that require chaining reasoning steps over domain knowledge — the kind of work that shows up in legal analysis, medical differential diagnosis, or financial modeling.

Hybrid architectures at scale. Teams use RAG and fine-tuning together, but there is no published playbook for how to evaluate a combined system. Measuring whether the fine-tuned component or the retrieval component is the failure point in a hybrid system requires instrumentation that most teams do not have in place early in a project.

If you are running production comparisons and have data on these questions, the community would benefit from more published case studies with actual numbers.