When model output disappoints, teams tend to assume they need to train something. In practice the fix is usually cheaper. The right mental model: prompting changes instructions, retrieval changes knowledge, and fine-tuning changes behavior. Pick the lever that matches the problem.
Exhaust prompting first
Prompt changes ship in minutes and cost nothing beyond tokens. Clearer instructions, better few-shot examples, and structured output constraints resolve a surprising share of quality complaints. If the model clearly knows how to do the task but does it inconsistently, keep working the prompt before doing anything else.
Use retrieval for knowledge gaps
If the model gives outdated or invented facts about your domain, no amount of fine-tuning fixes that reliably. Retrieval-augmented generation feeds current, authoritative content into the prompt at request time and lets you update knowledge by updating documents.
Fine-tune to change behavior at scale
Fine-tuning earns its cost in specific situations:
- You need a consistent voice, format, or classification scheme across millions of requests.
- Your prompt has grown enormous with examples, and baking them in would cut latency and cost.
- A small, cheap model could replace an expensive one if it were specialized for your task.
The prerequisite is data: hundreds to thousands of high-quality input-output pairs that represent exactly the behavior you want. Weak training data produces a model that is confidently wrong in your own house style.
A simple decision path
Wrong facts: add retrieval. Wrong behavior, low volume: improve the prompt. Wrong behavior at high volume with good training data available: fine-tune. Re-evaluate whenever a new base model ships, because a better foundation regularly erases the gains of last year's fine-tune.