Skip to main content

Prompting, RAG or fine-tuning: which fix fits your AI agent?

Match the fix to the failure: prompting for instructions, RAG for missing or changing knowledge, fine-tuning for behaviour prompts cannot hold. Evals decide.

Share -
An engineer seen from behind sits at a desk with several monitors showing code and a glowing blue AI brain model, with rows of lit server racks in a dark data centre beyond.

Choose by the failure you are fixing. If the agent misreads instructions or format, improve the prompt and context. If it lacks private or changing knowledge, add retrieval (RAG). If it must behave consistently in ways prompts cannot hold, consider fine-tuning. Most production agents combine prompting and RAG, and only some need fine-tuning.

Start from the failure, not the technique

Teams often treat prompting, retrieval and fine-tuning as a ladder: try prompts, then RAG, then fine-tune when all else fails. In practice they fix different problems, and climbing the ladder by default wastes effort. A fine-tuned model still cannot answer from a policy document it has never seen, and a better prompt will not teach a model last week's price list.

So the first step is to write down what is going wrong, with real examples. Collect the cases where the agent gave a poor answer or took the wrong action, and sort them:

  • Instruction failures. The agent ignores a rule, uses the wrong format or tone, or skips a step it was told to take.
  • Knowledge failures. The agent does not know a fact, uses an out-of-date one, or invents something it should have looked up.
  • Behaviour failures. The agent follows the rules most of the time but drifts on a narrow, repeated task, such as classifying requests into your own categories or writing in a strict house format.

Each group points to a different fix.

When better prompting and context is enough

Instruction failures are usually fixed in the prompt and the context around it. Clear instructions, a short set of worked examples, well-described tools and only the information the step needs go a long way. Our guide to context engineering for AI agents covers how to decide what the model sees at each step.

Prompt changes are cheap to make and easy to roll back, which is why they come first. They still need discipline: version them, test them against the same cases each time and put them through change approval before they reach production.

When the agent needs retrieval (RAG)

Knowledge failures call for retrieval. RAG fetches the relevant passages from your documents, tickets or records at the moment of the request and gives them to the model alongside the question. It suits knowledge that is private, changes often or must be traceable to a source.

RAG also helps with governance. Because each answer is built from retrieved material, you can log what was used, restrict what each user is allowed to see and update a fact by changing one document instead of retraining a model. The weak point is usually retrieval itself, so test retrieval before the model and plan how documents are refreshed and re-indexed when they change.

When fine-tuning earns its cost

Fine-tuning changes how a model behaves, not what it can look up. It is worth considering when a narrow task repeats at volume and the model keeps drifting despite good prompts and examples, or when a smaller model needs to match a larger one on one job to cut cost or response time.

It brings work that prompting and RAG do not:

  • Training data. You need a clean, labelled set of good examples, and someone must own its quality.
  • Retraining. Each new base model or change in requirements may mean training again.
  • Hosting and access. Some providers limit which models can be tuned, and a tuned model may need to run on your own infrastructure, as covered in our guide to running an AI agent on a self-hosted model.
  • Facts go stale. Knowledge trained into weights cannot be traced or updated cleanly, so keep facts in retrieval even when behaviour is tuned.

How the three combine in production

Most production agents use prompting and RAG together: the prompt sets the rules and the retrieved context supplies the facts. Fine-tuning, where it is used, sits underneath for one or two narrow behaviours, while retrieval still carries the knowledge. Keeping the roles separate makes each part easier to test and change.

Let evaluations make the call

Whichever fix you try, measure it against a fixed set of real cases before and after. Without that, a change that helps one example can quietly break others. Agent evaluations show whether a prompt change, a retrieval change or a tuned model actually improved results, and they tell you when the simpler fix was enough.

If you want help working out which of these your agent needs, talk to our team.

Frequently asked questions

Should we fine-tune a model on our company documents?

Usually not. Fine-tuning on documents tends to teach style rather than reliable facts, and the knowledge cannot be traced to a source or updated without training again. Retrieval is a better fit for company documents because it fetches the current text at request time and lets you show and log where an answer came from. Fine-tune only if a repeated behaviour still drifts after prompting and retrieval are working well.

Is RAG still needed now that models accept very long context?

Often, yes. Long context lets you pass more material in one request, but sending everything each time raises cost and response time, and models can miss details buried in a large input. Retrieval selects the passages that matter, applies access rules for each user and records what was used. Long context is useful for a single large document, while retrieval suits a knowledge base that grows and changes.

How do we know when prompting has reached its limit?

Prompting has reached its limit when the same failure keeps returning across many test cases after several careful prompt revisions, and the cause is not missing knowledge or a badly described tool. If adding examples fixes some cases but breaks others, and the task is narrow and frequent, fine-tuning may be worth testing. Keep the evaluation set fixed so you can compare the tuned model fairly against the prompted one.

Can we switch base models after fine-tuning?

Not directly. A fine-tune is tied to the base model it was trained on, so moving to a newer model usually means training again with the same data. That is one reason to keep training data, evaluation cases and prompts versioned and owned by the team. Agents built mainly on prompting and retrieval move between models more easily, because the knowledge and rules live outside the model weights.

Share this post

Contact us to build the right product

Talk to our engineers about your application, the systems it connects to, and what you want to build next.

Get in touch
Two people discussing work with a laptop