Evaluation
A fine-tuned model is only better if it improves the target task without creating unacceptable regressions elsewhere.
A fine-tuned model is only better if it improves the target task without creating unacceptable regressions elsewhere.
Measure the behavior you trained for.
Check important old capabilities.
Inspect qualitative failures.
Understand the working model
Keep evaluation examples separate from training data. If the model already saw the exact examples during training, a high score may reflect memorization rather than generalization.
Measure what users care about: correctness, format compliance, tone, refusal behavior, latency or cost. One aggregate score rarely captures every important tradeoff.
Practice
Apply the pattern above to one real task you already do. Keep the first version small enough that you can inspect every input and output.
Do not select a model using examples that were used to tune the same model.
Use held-out evaluation, regression checks and human review before deciding that fine-tuning helped.