LLM & RAG Evaluation Suite

Multi-Model Chatbot Benchmarking, Retrieval-Augmented Generation (RAG) Auditing & LLM-as-a-Judge Tracing

An enterprise-grade LLM evaluation workspace applying LangSmith tracing and LLM-as-a-Judge grading patterns to benchmark conversational assistants and Retrieval-Augmented Generation (RAG) pipelines. Features automated dataset creation, multi-provider execution across Google Gemini, Groq (Llama 3.3), and OpenAI, and structured grading schemas for factual correctness, concision, response relevance, groundedness (hallucination detection), and retrieval quality.

---

How It Works

This repository implements a rigorous evaluation methodology built on LangSmith experiment tracking and structured LLM-as-a-Judge grading pipelines. Each module evaluates language model performance across distinct operational tiers, measuring both end-to-end chatbot responsiveness and multi-step RAG retrieval fidelity.

View LLM & RAG Evaluation Architecture Flow Diagram

graph TD
    classDef main fill:#1C3C3C,stroke:#333,stroke-width:1px,color:#fff;
    classDef llm fill:#3776AB,stroke:#333,stroke-width:1px,color:#fff;
    classDef eval fill:#F55036,stroke:#333,stroke-width:1px,color:#fff;
    classDef rag fill:#008080,stroke:#333,stroke-width:1px,color:#fff;
    classDef store fill:#f4f4f4,stroke:#333,stroke-width:1px,color:#333;

    Workspace([Evaluation Dataset / Test Cases]) --> Route{Evaluation Pipeline}
    
    %% Pipeline 1: Chatbot Benchmarking
    Route -->|Direct Chatbot QA| P1[Chatbot Target Application]:::llm
    P1 -->|Gemini 2.5 Flash / 2.0 Lite| Res1["Model Response Generation"]:::llm
    P1 -->|Groq Llama-3.3-70B| Res1
    Res1 --> Judge1["LLM-as-a-Judge Evaluator"]:::eval
    Judge1 -->|Correctness & Concision| Metrics1([LangSmith Experiment Report])

    %% Pipeline 2: RAG Evaluation Triad
    Route -->|Knowledge Base Auditing| P2[RAG Evaluation Pipeline]:::rag
    P2 --> Loader["WebBaseLoader & RecursiveSplitter"]:::store
    Loader --> Store["InMemoryVectorStore (OpenAI Embeddings)"]:::store
    Store --> Retriever["Top-K Context Retriever"]:::rag
    Retriever --> RAGGen["RAG Answer Synthesizer"]:::rag
    
    RAGGen --> RAGJudge["Structured Judge Schema (Pydantic / TypedDict)"]:::eval
    RAGJudge -->|Factual Correctness| EvalCorrect["Correctness Grader"]:::eval
    RAGJudge -->|Hallucination Audit| EvalGround["Groundedness Grader"]:::eval
    RAGJudge -->|Context Accuracy| EvalRet["Retrieval Relevance Grader"]:::eval
    
    EvalCorrect & EvalGround & EvalRet --> RAGReport([Comprehensive RAG Scorecard])

Loading

Multi-Tiered Evaluation & Metric Guardrails

To prevent LLM regressions, hallucinations, and retrieval degradation, the suite enforces structured evaluation gates across all benchmark runs:

graph TD
    classDef safety fill:#F55036,stroke:#333,stroke-width:1px,color:#fff;
    classDef pipeline fill:#3776AB,stroke:#333,stroke-width:1px,color:#fff;
    classDef metric fill:#008080,stroke:#333,stroke-width:1px,color:#fff;

    Input([Test Query & Ground Truth]) --> Exec[Candidate LLM / RAG Execution]:::pipeline
    
    Exec --> CorrectGate{1. Factual Correctness Gate}:::safety
    CorrectGate -->|Semantic Match vs Reference| ConcGate{2. Concision & Style Gate}:::safety
    ConcGate -->|Length & Brevity Check| GroundGate{3. Groundedness / Hallucination Gate}:::safety
    GroundGate -->|Strict Context Attribution| RetGate{4. Retrieval Relevance Gate}:::safety
    
    RetGate --> Scorecard([LangSmith Trace & Experiment Score]):::metric

Loading

🛡️ Evaluation Metric Mappings

1. Factual Correctness Gate (correctness)

   • Mechanism: An expert LLM judge evaluates predicted answers against ground-truth reference outputs using structured grading prompts.
   • Boundary: Flags semantic deviations, contradictory claims, or inaccurate problem resolutions across conversational intents.

2. Concision & Style Gate (concision)

   • Mechanism: Deterministically measures output bloat against reference expectations (verifying response brevity $< 2\times$ reference length).
   • Boundary: Ensures customer-facing bots remain succinct and actionable without verbosity.

3. Groundedness & Hallucination Gate (groundedness)

   • Mechanism: Inspects synthesized answers strictly against retrieved vector store documents via structured schemas (GroundedGrade).
   • Boundary: Detects external or fabricated facts not supported by retrieved context.

4. Retrieval Relevance Audit (retrieval_relevance)

   • Mechanism: Evaluates whether documents retrieved from InMemoryVectorStore contain semantic relevance to the input query (RetrievalRelevanceGrade).
   • Boundary: Isolates embedding/chunking failures from generation failures.

---

Key Features

• Multi-Provider LLM Benchmarking — Automated comparative evaluation across Google Gemini 2.5 Flash, Gemini 2.0 Flash Lite, and Groq Llama 3.3 70B Versatile with built-in retry and fallback controllers.
• Structured LLM-as-a-Judge Grading — Enforces JSON schema outputs (TypedDict + structured tool calling) where the judge must explicitly write out reasoning explanations before assigning boolean scores.
• Complete RAG Triad Auditing — End-to-end evaluation covering retrieval relevance, response groundedness (hallucination prevention), and answer correctness.
• Automated Dataset Management — Programmatically provisions test datasets, multi-intent examples (flight_booking, refund_policy, math, escalation), and real-time execution traces in LangSmith.
• Resilient API Execution — Automatic fallback logic (call_with_fallback) that seamlessly switches from primary models to secondary inference endpoints on rate limits or API timeouts.

Tech Stack

Layer	Technology	Description
Evaluation Platform		Tracks datasets, experiment prefixes, execution traces, and custom evaluator metrics.
Orchestration Framework		Provides document loaders (`WebBaseLoader`), text splitters, vector stores, and structured prompt wrappers.
Google Gemini Models		Primary high-speed evaluation target and fallback inference evaluator via OpenAI API compatibility layer.
Groq Inference Engine		Ultra-low latency open-weights evaluation target (`llama-3.3-70b-versatile`).
Structured Judge Models		Structured JSON schema grading (`gpt-4o`, `gpt-4o-mini`) for strict RAG evaluation metrics.

Project Structure

.
├── Chatbot-evaluation/
│   ├── Evaluation.py            # Multi-provider chatbot benchmarking script & LangSmith runner
│   ├── Evaluation.ipynb         # Interactive notebook for chatbot evaluation & metric traces
│   ├── rag_evaluation.ipynb     # Comprehensive RAG Triad evaluation notebook
│   └── RAG-Evaluation.ipynb     # Retrieval experiment benchmarking suite
├── main.py                      # Global application entrypoint
├── pyproject.toml               # Modern UV project configuration & dependency specifications
├── requirements.txt             # Standard pip requirements file
├── .env                         # API credentials & LangSmith configuration
└── README.md                    # Evaluation suite documentation

Getting Started

Prerequisites

• Python 3.13+
• A LangSmith API Key for experiment tracking
• API Keys for target inference providers (Google Gemini, Groq, and OpenAI)

Installation

# Clone the repository
git clone https://github.com/your-username/llm-evaluation.git
cd llm-evaluation

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies using uv (recommended)
uv pip install -r requirements.txt
# Or using standard pip:
pip install -r requirements.txt

Configuration

Create a .env file in the root directory to configure your tracing and model provider credentials:

# LangSmith Observability
LANGSMITH_API_KEY=your_langsmith_api_key_here
LANGSMITH_TRACING=true

# Inference Providers
GEMINI_API_KEY=your_gemini_api_key_here
GROQ_API_KEY=your_groq_api_key_here
OPENAI_API_KEY=your_openai_api_key_here

---

The Evaluation Workflows

1. Multi-Model Chatbot Benchmarking

Located in Chatbot-evaluation/Evaluation.py and Evaluation.ipynb, this workflow creates a multi-intent test suite (Chatbot evaluation) covering flight bookings, refund policies, reasoning tasks, and customer escalations.

graph LR
    classDef llm fill:#3776AB,stroke:#333,stroke-width:1px,color:#fff;
    classDef eval fill:#F55036,stroke:#333,stroke-width:1px,color:#fff;

    Data[LangSmith Dataset] --> TargetGemini[Gemini 2.5 Flash / 2.0 Lite]:::llm
    Data --> TargetGroq[Groq Llama 3.3 70B]:::llm
    
    TargetGemini & TargetGroq --> Judge[LLM-as-a-Judge Grader]:::eval
    Judge --> CorrectnessCheck[Correctness Score]
    Judge --> ConcisionCheck[Concision Score]

Loading

Tip

Resilient Multi-Provider Fallbacks The evaluation harness wraps model invocations in call_with_fallback, ensuring that if a provider experiences temporary rate limits or quota exhaustion, evaluation automatically falls back to secondary high-capacity models without interrupting benchmark runs.

2. Retrieval-Augmented Generation (RAG) Triad Auditing

Located in Chatbot-evaluation/rag_evaluation.ipynb, this workflow evaluates a production RAG pipeline built over web documentation using WebBaseLoader and InMemoryVectorStore.

graph TD
    classDef rag fill:#008080,stroke:#333,stroke-width:1px,color:#fff;
    classDef judge fill:#F55036,stroke:#333,stroke-width:1px,color:#fff;

    Query([User Question]) --> Retriever[Vector Store Retrieval]:::rag
    Retriever --> Docs[Retrieved Chunk Context]
    Docs & Query --> Generator[RAG Synthesizer]:::rag
    
    Generator --> Answer[Generated Answer]
    
    Query & Docs --> RelJudge["1. Retrieval Relevance Grader"]:::judge
    Docs & Answer --> GroundJudge["2. Groundedness / Hallucination Grader"]:::judge
    Query & Answer & GroundTruth --> AccJudge["3. Correctness Grader"]:::judge

Loading

Important

Structured Output Grading Schemas Every RAG evaluator enforces strict JSON schema generation where the model must generate an explanation rationale before assigning its final boolean assessment (correct, grounded, or relevant). This step-by-step reasoning significantly increases grading accuracy and auditability.

class CorrectnessGrade(TypedDict):
    explanation: Annotated[str, ..., "Explain your reasoning for the score"]
    correct: Annotated[bool, ..., "True if the answer is correct, False otherwise."]

---

Customization

The evaluation suite is designed for modular customization across models, datasets, and custom judges:

Component	Target File	Modification Details
Add New Evaluation Metrics	Evaluation.py	Define custom grading functions (e.g., toxicity_evaluator or tone_evaluator) and pass them to client.evaluate(evaluators=[...]).
Switch Vector Store / Embeddings	rag_evaluation.ipynb	Swap InMemoryVectorStore with persistent vector databases (Qdrant, Chroma, Pinecone) or custom embedding models.
Customize Test Datasets	Evaluation.py	Modify input/output dictionaries inside client.create_examples(...) to evaluate domain-specific medical, legal, or financial support bots.
Adjust LLM Judge Strictness	rag_evaluation.ipynb	Customize grading criteria inside correctness_instructions, grounded_instructions, or retrieval_relevance_instructions.

---

Acknowledgments

• LangSmith Observability & Evaluation for dataset and trace orchestration.
• LangChain Core & Community for RAG components and document loaders.
• Lilian Weng's Technical Blog for benchmark reference datasets and AI agent research.