Why standard metrics fail for RAG
BLEU and ROUGE measure n-gram overlap against a reference answer. In a RAG system there is often no single correct reference: a question about company policy may have dozens of valid phrasings. High BLEU does not mean the system cited the right source; low BLEU does not mean it was wrong.
The three metrics that actually matter
Faithfulness measures whether every claim in the generated answer can be traced back to a retrieved passage. A faithfulness score of 1.0 means the model invented nothing. Tools like RAGAS implement this with an LLM judge.
Context Precision asks: of the passages retrieved, how many were actually relevant to the question? Low precision wastes context window and increases hallucination risk.
Answer Relevancy checks whether the final response actually addresses what was asked, not just whether it sounds good.
Building an eval harness
Start with a golden dataset: 100-200 question/answer pairs that domain experts have verified. Run your pipeline against them nightly. Track the three metrics above over time. A drop in Faithfulness after a model upgrade is a red flag; a drop in Context Precision after a chunking change means your retrieval is degrading.
The harness does not have to be complex. A spreadsheet with automatic scoring via the OpenAI or Anthropic API is enough to start catching regressions before they reach production.