When should you fine-tune?
Before using training compute, decide whether the problem is instructions, knowledge or repeated behavior.
Change instructions at run time.
Add current external knowledge.
Change learned behavior patterns.
Train a smaller adapter instead of every parameter.
Fine-tuning is not the first fix
If the problem is missing facts, use retrieval. If the problem is an unclear instruction, improve prompting. Fine-tuning becomes more relevant when you need a repeated behavior, style or task pattern that examples can demonstrate consistently.
Do not fine-tune to “upload knowledge.”
Frequently changing facts are usually a better fit for retrieval or another external knowledge mechanism.
Choose the smallest intervention that solves the problem: prompting first, retrieval for knowledge, fine-tuning for repeatable learned behavior.