Dynamic RAG for Technical Standards Validation

AI-powered document intelligence system that helps engineers identify whether technical standards are still valid, active, relevant, or superseded using Retrieval-Augmented Generation (RAG) and semantic search. Built for offline environments with complete privacy and security.

Problem

Organizations often maintain hundreds of technical standards and reference documents. Engineers must manually verify:

• Whether a document is still active or deprecated
• Whether a newer version exists
• Whether a standard remains relevant to current requirements
• Whether references and dependencies are still valid
• How changes in parent standards cascade to internal technical standards

This manual review process is time-consuming, error-prone, and difficult to scale as standards libraries grow.

Why Existing Search Failed

Traditional document retrieval approaches rely heavily on:

• Exact keyword matching
• Manual categorization
• Human review processes

These methods struggle when:

• Similar concepts use different terminology, abbreviations, or document structures
• Multiple document versions exist without clear version tracking
• References are scattered across standards documents
• Relevance depends on semantic meaning rather than exact terms
• Impact analysis requires understanding implicit relationships between documents

Keyword-based search simply cannot handle the semantic complexity of technical standards validation at scale.

Solution Architecture

The system combines semantic retrieval, relationship mapping, and LLM reasoning to automate standards validation:

1. Document Ingestion: Technical standards and reference documents are parsed and chunked for analysis
2. Semantic Embedding: Text chunks are converted to vector embeddings using local embedding models
3. Vector Indexing: Embeddings are indexed using FAISS for efficient similarity search
4. Relationship Mapping: Standards relationships are tracked in a knowledge graph (SUPERSEDES, REFERENCES, HARMONIZED_WITH)
5. Semantic Retrieval: User queries retrieve semantically similar standards through vector similarity
6. LLM Reasoning: Retrieved context is analyzed by an LLM to produce structured assessments
7. Explainable Output: The system provides reasoning traces showing why documents are active, inactive, or superseded
8. Impact Analysis: Change detection simulates how updates cascade through the standards dependency graph

Architecture

Tech Stack

• Language: Python 3.10+
• Web Framework: FastAPI
• Vector Search: FAISS (CPU-based, offline)
• Local LLM: Ollama (with configurable models)
• Embeddings: nomic-embed-text (local)
• Data Validation: Pydantic
• Frontend: Vanilla HTML/CSS/JavaScript
• Key Feature: 100% offline operation — no external API calls, no cloud dependencies

Key Features

• Semantic Document Retrieval: Finds relevant standards even when using different terminology
• Standards Validity Verification: Determines if documents are active, inactive, or superseded
• Relationship Mapping: Tracks SUPERSEDES, REFERENCES, HARMONIZED_WITH relationships
• Impact Analysis: Simulates how changes in parent standards cascade through dependent documents
• Vendor Question Generation: Automatically generates targeted evaluation questions
• Explainable Reasoning: Provides structured assessment reasoning (not just predictions)
• Privacy-First Design: 100% offline operation with zero external API calls
• Interactive Dashboard: Web UI for querying, analyzing, and managing standards

Getting Started

Prerequisites

1. Python 3.10 or higher
2. Ollama installed on your system (Download Ollama)

Installation

Clone the repository and install dependencies:

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install Python dependencies
pip install -r requirements.txt

Setup Local Models

Pull the required Ollama models:

# Generation model (default: gpt-oss:20b, alternatives: llama3, mistral, etc)
ollama pull gpt-oss:20b

# Embedding model (required for semantic retrieval)
ollama pull nomic-embed-text

Running the Application

Start the FastAPI backend server:

python -m app.main

The server will initialize the FAISS index, load standards data, and start the web interface.

Access the application:

• Dashboard: http://localhost:8000/
• API Documentation: http://localhost:8000/docs

Demo

The demo shows:

• Semantic search across technical standards
• Real-time standards assessment
• Impact analysis of standard updates

Project Structure

Dynamic-RAG/
├── app/
│   ├── api.py                  # FastAPI route definitions
│   ├── main.py                 # Application entry point & component initialization
│   ├── llm/
│   │   └── ollama_client.py    # Local inference client handling generation & embeddings
│   ├── models/
│   │   └── schemas.py          # Pydantic data models used across the system
│   ├── rag/
│   │   ├── embeddings.py       # Wrapper for embedding generation strategy
│   │   └── retriever.py        # FAISS vector store integration
│   ├── reasoning/
│   │   ├── compliance_reasoner.py # Logic for evaluating compliance
│   │   ├── governance.py          # AI guardrails & Vendor Question Generator
│   │   ├── impact_analysis.py     # Predicts updates cascading effects
│   │   ├── its_draft_generator.py # Generates technical standard drafts
│   │   └── refiner.py             # Iterative improvement of recommendations
│   ├── standards/
│   │   ├── change_detection.py    # Engine to simulate/track standard document changes
│   │   ├── knowledge_graph.py     # Graph mappings for standard relationships
│   │   ├── template_manager.py    # Manages standard templates
│   │   └── versioned_store.py     # In-memory storage of standard metadata
│   ├── evaluation/
│   │   ├── competitive_baselines.py # Comparison with other retrieval methods
│   │   ├── evaluation_controller.py # Evaluation orchestration
│   │   ├── lifecycle_evaluation.py  # Standards lifecycle evaluation
│   │   └── run_lifecycle_evaluation.py # Evaluation runner
│   └── static/
│       ├── index.html          # Main dashboard
│       └── admin.html          # Admin interface
├── requirements.txt            # Python dependencies
└── README.md                   # Project documentation

Results

The framework demonstrates how RAG can support document governance workflows by:

• Reducing Manual Review Effort: Automating the identification of document validity and relevance
• Improving Retrieval Quality: Semantic search significantly outperforms keyword-based matching for standards discovery
• Supporting Evidence-Based Decisions: Retrieved context provides clear reasoning for validation assessments
• Enabling Impact Analysis: Quickly identifies which internal standards require updates when parent standards change
• Scaling Standards Management: Handles hundreds of standards documents with consistent, explainable outputs

Lessons Learned

During development, several key insights emerged:

Retrieval Quality Depends on Chunking Strategy

• Document chunking significantly impacts what the system can retrieve
• Fixed-size chunks sometimes break semantic units, while semantic chunking requires domain knowledge
• The choice affects both retrieval accuracy and LLM context window utilization

Document Metadata is Critical

• Metadata (version, date, status, relationships) strongly influences retrieval accuracy
• Missing or inconsistent metadata makes semantic similarity less effective
• Standards libraries need structured metadata to support automated validation

LLM Reasoning is Limited by Retrieved Context

• The quality of the final assessment is bounded by what was retrieved
• A single missed relevant standard can lead to incorrect validity assessments
• Explainability requires showing both retrieved documents and reasoning steps

Governance Workflows Demand Explainability

• Standards validation requires clear reasoning, not just predictions
• Engineers must understand why a standard is deemed active, inactive, or superseded
• Black-box recommendations are insufficient for compliance decisions

These findings informed the final system architecture, prioritizing transparency and evidence-based reasoning over raw accuracy metrics.

Governance & Compliance

This tool acts as a Decision Support System. Built-in governance mechanisms explicitly add disclaimers to AI outputs:

This assessment is based on standard metadata only. Always consult the full standard text for definitive compliance requirements. Human expert verification is REQUIRED before implementing any recommendations.