Measuring RAG Quality: RAGAS Metrics, Answer Relevance, and Catching Hallucinations

Your RAG chatbot feels good in demos. You have no idea if it's hallucinating 5% of the time or 30% of the time. RAGAS tells you exactly. You’ve cobbled together a pipeline: some chunking, text-embedding-3-large, a vector store, and GPT-4o. It answers questions about your internal docs. In your terminal, it looks like magic. In production, it’s a black box spitting out plausible-sounding nonsense at a rate you can only guess at. Vibes are not a metric. Your product manager’s "seems okay" is not a SLA. We need numbers.

Why Your Gut Is a Terrible RAG Evaluator

You demo with the same three golden queries you built the system around. Of course it works. You’re not testing it; you’re performing a ritual. The moment a user asks something slightly orthogonal—or the retrieval fetches a vaguely similar but incorrect chunk—your LLM, trained to be helpful, confidently stitches together an answer from fragments of truth and imagination. Meta AI Research 2025 found that while RAG slashes the LLM hallucination rate from 27% to 6% on domain-specific queries, that 6% isn’t a flat tax. It’s a landmine in your less-traveled data paths.

The cost of being wrong isn’t just a confused user. It’s regulatory risk, eroded trust, and burning cash. GPT-4o processing 1 trillion tokens a month means a lot of people are paying for answers. At an average cost of $5 per million tokens, those hallucinations are expensive fabrications. You can’t A/B test your way to reliability by staring at logs. You need an automated, quantitative evaluation framework that runs before you ship. Your intuition scales to about ten examples. Your user base does not.

The RAGAS Trinity: Faithfulness, Answer Relevance, Context Recall

RAGAS (Retrieval-Augmented Generation Assessment) isn’t a tool; it’s a wake-up call. It breaks down the monolithic "is this answer good?" into three measurable failure points. Think of it as distributed tracing for your RAG pipeline.

1. Faithfulness: Is the generated answer grounded exclusively in the provided context? This is your hallucination detector. A score of 0.95 means 95% of the factual statements in the answer can be attributed to the retrieved chunks. The other 5% is your LLM inventing facts.
2. Answer Relevance: Is the answer directly useful to the question? An answer can be perfectly faithful but utterly irrelevant. If the question is "How do I reset the database password?" and the context is about database architecture, a faithful summary of that architecture is irrelevant. This metric penalizes verbose, evasive, or off-topic responses.
3. Context Recall: Did the retriever fetch all the relevant context needed to answer the question? This evaluates your retrieval engine, not your LLM. If the gold answer requires information in chunks A, B, and C, but you only retrieved A and B, your context recall is ~0.66. This is where your embedding model and chunking strategy get graded.

These metrics are LLM-evaluated themselves. RAGAS uses a lightweight model (like GPT-3.5-Turbo) to judge the outputs of your heavier, more expensive pipeline model (like GPT-4o). It’s a cost-effective supervisor.

Building Your Evaluation Dataset: Synthetic QA vs. The Blood, Sweat, and Tears Method

You can’t evaluate what you can’t measure against. You need a benchmark dataset of (question, context, answer) triples.

Option 1: Synthetic Generation (The Scalable Hack) Use a good LLM to generate questions from your documents. It’s fast and covers broad ground. With instructor, you can structure this cleanly.

import instructor
from openai import OpenAI
from pydantic import BaseModel
from typing import List

client = instructor.patch(OpenAI())

class QAPair(BaseModel):
    question: str
    answer: str  # The LLM's *ideal* answer based on the chunk

def generate_qa_from_chunk(chunk: str, model="gpt-4o-mini") -> List[QAPair]:
    """Generate 2-3 QA pairs from a single text chunk."""
    prompt = f"""
    Based on the following context, generate 2-3 question-answer pairs.
    Questions should be diverse: factual, procedural, and conceptual.
    Answers must be grounded solely in the provided context.

    Context:
    {chunk}
    """
    response = client.chat.completions.create(
        model=model,
        response_model=List[QAPair],
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
    )
    return response


chunk = "Our API rate limit is 1000 requests per minute per API key. Exceeding this limit results in a RateLimitError. Implement exponential backoff using the `tenacity` library for robust retries."
qa_pairs = generate_qa_from_chunk(chunk)
print(qa_pairs[0].question)  # "What happens if I exceed 1000 RPM on the API?"

Option 2: Human-Labeled (The Ground Truth) This is the gold standard. Have SMEs (Subject Matter Experts) write questions and extract answers from documents. It’s slow, expensive, and non-negotiable for high-stakes domains (legal, medical). Start with 50-100 high-quality pairs for critical validation. Use synthetic data for breadth, human data for depth and validation of your synthetic process.

Running RAGAS: From Setup to Scorecard

Let’s move from theory to a runnable evaluation. We’ll use a simple pipeline with LlamaIndex and evaluate it with RAGAS.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings.openai import OpenAIEmbedding
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevance, context_recall
from datasets import Dataset
import os

# 1. Build a simple index (your existing RAG)
os.environ["OPENAI_API_KEY"] = "your-key"
documents = SimpleDirectoryReader("./your_docs").load_data()
embed_model = OpenAIEmbedding(model="text-embedding-3-large")
index = VectorStoreIndex.from_documents(documents, embed_model=embed_model)
query_engine = index.as_query_engine(similarity_top_k=3) # Retrieve top 3 chunks

# 2. Define your evaluation dataset (e.g., 5 human-labeled examples)
eval_questions = [
    "What is the procedure for resetting a user's MFA?",
    "What is the SLA for API response time P99?",
]
ground_truth_answers = [
    "The procedure is documented in the IT portal and requires manager approval.",
    "The P99 API response time SLA is 250 milliseconds.",
]
# For context_recall, you need the *ideal* retrieved contexts. Often approximated by the source documents for the answer.

# 3. Run your RAG pipeline on the eval questions
answers = []
contexts_list = []
for question in eval_questions:
    response = query_engine.query(question)
    answers.append(response.response)
    # Extract the retrieved context nodes
    contexts_list.append([node.text for node in response.source_nodes])

# 4. Format for RAGAS evaluation
data_dict = {
    "question": eval_questions,
    "answer": answers,
    "contexts": contexts_list,
    "ground_truth": ground_truth_answers
}
dataset = Dataset.from_dict(data_dict)

# 5. Run the evaluation
result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevance, context_recall],
    llm="gpt-3.5-turbo-16k",  # The judge model
    embeddings="text-embedding-3-small", # For some metrics
)
print(result)
df = result.to_pandas()
print(df[['faithfulness', 'answer_relevance', 'context_recall']].mean())

The output is a pandas DataFrame with a score for each metric per example, plus averages. A score below 0.8 in any category warrants investigation.

Debugging Low Scores: Is It Your Retriever or Your Generator?

When your RAGAS report looks grim, you need to diagnose where the pipeline broke.

Low Faithfulness (< 0.85): Your LLM is hallucinating.

• Fix: Strengthen your system prompt. "You are a precise assistant. Answer ONLY using the provided context. If the context does not contain sufficient information to answer the question, say 'I cannot answer based on the provided information.' Do not extrapolate or infer." Enforce this with response_format or libraries like guidance.
• Real Error & Fix: Hallucination in factual query. Implement retrieval verification: have a secondary, cheaper LLM call check each factual claim in the answer against the context before it's sent to the user.

Low Answer Relevance (< 0.8): Your answer is off-topic or verbose.

• Fix: This is often a prompt engineering issue. Add instructions like "Provide a concise, direct answer to the question." Also, check if your retriever is fetching completely irrelevant context, forcing the LLM to ramble.

Low Context Recall (< 0.7): Your retriever is missing key pieces.

• Fix: This is a retrieval problem. Look at your similarity_top_k. Are you retrieving enough chunks? Is your chunk size appropriate? Compare your embedding model.

If your dense embeddings are underperforming, you might be hitting a ContextWindowExceededError during chunking. Fix: Implement a smarter chunking strategy (semantic chunking with overlap using LangChain's RecursiveCharacterTextSplitter or markdown-aware splitting) before embedding.

Shipping with Confidence: Continuous RAG Evaluation in CI/CD

Evaluation isn’t a one-off. Integrate it into your CI/CD pipeline to block regressions.

# .github/workflows/evaluate-rag.yaml
name: Evaluate RAG Pipeline
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install ragas llama-index openai datasets
      - name: Run RAGAS Evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python scripts/run_ragas_eval.py
      - name: Check Metric Thresholds
        run: |
          python scripts/check_metrics.py

Your check_metrics.py script should load the results and fail the build if thresholds aren’t met:

import pandas as pd
import sys

df = pd.read_csv('ragas_results.csv')
avg_faithfulness = df['faithfulness'].mean()

if avg_faithfulness < 0.85:
    print(f"❌ Faithfulness score {avg_faithfulness} below threshold of 0.85. Blocking deployment.")
    sys.exit(1)
else:
    print(f"✅ Faithfulness score {avg_faithfulness} acceptable.")

This is your safety net. A new embedding model, a tweak to the chunk size, or a prompt "optimization" that introduces hallucinations will now break the build instead of your production system.

Going Beyond Defaults: Crafting Domain-Specific Metrics

RAGAS provides the foundation, but your domain has unique needs. You need custom metrics.

• Citation Accuracy: For legal or academic RAG, where the answer comes from is as important as the answer itself. Build a metric that checks if the generated answer correctly cites the retrieved document IDs.
• Toxicity/Safety: If your RAG interacts with users, add a safety metric using a classifier like perspective-api or a local model to catch any toxic output, even if it's faithful.
• Latency & Cost: These are operational metrics, but they belong in your evaluation suite. If your new "improved" pipeline increases average latency from 800ms to 1200ms or doubles token consumption, you need to know. Integrate with Helicone or Arize to track these alongside quality.

Real Error & Fix: JSON mode output not valid JSON. When using custom metrics that require structured output, always wrap the call. Fix: Use response_format={'type':'json_object'} in the OpenAI API and wrap parsing in a try/except with a retry fallback.

Next Steps: From Evaluation to Observability

You now have a number. 0.92 faithfulness. Great. But that’s a snapshot. Production is a movie. Your next investment is LLM Observability.

1. Instrument Your Pipeline: Use LangSmith or Weights & Biases to trace every query, retrieval, and generation. Log the inputs, outputs, contexts, and calculated RAGAS scores (or their proxies) for a sample of production traffic.
2. Set Up Alerting: When your real-time faithfulness proxy dips for a specific document cluster or user segment, you should get a PagerDuty alert before users do.
3. Continuous Dataset Curation: Use production logs to find hard examples—questions where confidence was low or user feedback was negative. Automatically add these to your evaluation dataset. Your system should get smarter with every failure.

RAG evaluation isn’t a task you finish. It’s a core discipline. It moves you from hoping your RAG works to knowing its failure modes, its costs, and its limits. Stop demoing. Start measuring. Your RTX 4090—and your users—will thank you.