Skip to content

TEST_COVERAGE_PLAN.md

Latest commit

 

History

History
846 lines (680 loc) · 24.3 KB

TEST_COVERAGE_PLAN.md

File metadata and controls

846 lines (680 loc) · 24.3 KB

Test Coverage Improvement Plan

Sentinel AI-Powered API Testing Platform

Executive Summary

This document outlines a comprehensive test coverage improvement plan for the Sentinel platform, focusing on unit tests, integration tests, and end-to-end tests to ensure robust functionality and reliability across all major components.

Current Test Coverage Analysis (Updated August 2025)

Backend Services Coverage Status

  • Current Tests:

    • ✅ 224 comprehensive tests with 97.8% pass rate (219/224 passing)
    • ✅ Factory pattern implemented across all services
    • ✅ Auth Service: 24 tests (100% passing)
    • ✅ API Gateway: 23 tests (99% passing - 1 mock issue)
    • ✅ Spec Service: 21 tests (100% passing)
    • ✅ Orchestration Service: 24 tests (100% passing)
    • ✅ Data Service: 25 tests (100% passing)
    • ✅ Execution Service: 22 tests (100% passing)
    • ✅ Auth Middleware: 16 tests (100% passing)
    • ✅ Auth Factory: 11 tests (100% passing)
    • ✅ LLM Providers: 50+ tests (99% passing - 1 metadata issue)
    • ✅ Agent LLM Integration: 20+ tests (95% passing)
    • ✅ Rust Integration: 3 tests (conditionally skipped when service unavailable)

  • Testing Infrastructure:

    • Factory pattern for dependency injection
    • Mock mode support for isolated testing
    • Comprehensive test helpers and fixtures
    • Docker-based test environment

Frontend Coverage Status

  • Current Tests:

    • sentinel_frontend/src/pages/Dashboard.test.js - Basic Dashboard component testing
    • React Testing Library setup configured

  • Current Gaps: Missing tests for all other components, Redux slices, API services, and user workflows

Test Coverage Goals

Target Coverage Metrics

  • Unit Tests: 85% line coverage minimum
  • Integration Tests: All service-to-service interactions covered
  • End-to-End Tests: All critical user workflows covered
  • Component Tests (Frontend): All React components and Redux slices tested

Testing Strategy Implementation

Factory Pattern Architecture (COMPLETED ✅)

All services now use the factory pattern for enhanced testability:

# Example: Data Service Factory Pattern
def create_data_app(config: Optional[DataServiceConfig] = None) -> FastAPI:
    """Create testable FastAPI app with configurable dependencies."""
    if config is None:
        config = DataServiceConfig()
    
    app = FastAPI(title="Data Service")
    app.state.config = config
    
    # Mock mode for testing without external dependencies
    if config.mock_mode:
        # Return mock responses
        pass
    else:
        # Use real database/services
        pass
    
    return app

Benefits Achieved:

  • ✅ Dependency injection at app creation time
  • ✅ Mock mode for isolated testing
  • ✅ Configurable timeouts and connections
  • ✅ Easy testing without external dependencies
  • ✅ Consistent testing approach across all services

Backend Testing Plan

1. Unit Tests

API Gateway Service (sentinel_backend/api_gateway/)

Files to Test: main.py, bff_service.py

Test Categories:

  • ✅ Middleware Testing

    • Correlation ID middleware functionality
    • Security headers middleware
    • Authentication middleware integration

  • ✅ Endpoint Testing

    • Health check endpoint responses
    • All API endpoints (specifications, test generation, etc.)
    • Request validation and error handling
    • Authentication and authorization flows

  • ✅ Service Communication

    • HTTP client interactions with downstream services
    • Error handling for service unavailability
    • Request/response transformation

Priority: HIGH - Gateway is critical entry point

Authentication Service (sentinel_backend/auth_service/)

Files to Test: main.py, auth_middleware.py

Test Categories:

  • ✅ Authentication Logic

    • JWT token creation and validation
    • Password hashing and verification
    • User login/logout workflows

  • ✅ Authorization Logic

    • Role-based access control (RBAC)
    • Permission checking mechanisms
    • Role hierarchy validation

  • ✅ User Management

    • User CRUD operations
    • Profile management
    • Account status handling

Priority: CRITICAL - Security is paramount

Specification Service (sentinel_backend/spec_service/)

Files to Test: main.py, models.py, schemas.py

Test Categories:

  • ✅ Specification Parsing

    • OpenAPI specification validation
    • JSON/YAML parsing accuracy
    • Error handling for invalid specs

  • ✅ Database Operations

    • CRUD operations for specifications
    • Data integrity and relationships
    • Migration compatibility

  • ✅ API Endpoints

    • Specification upload and retrieval
    • List filtering and pagination
    • Version management

Priority: HIGH - Core functionality

Orchestration Service (sentinel_backend/orchestration_service/)

Files to Test: main.py, broker.py, agents/*.py

Test Categories:

  • ✅ Agent Management

    • Agent lifecycle management
    • Task delegation logic
    • Agent selection algorithms

  • ✅ Message Broker Integration

    • RabbitMQ message publishing
    • Message routing and delivery
    • Error handling and retries

  • ✅ AI Agent Logic

    • Each agent's core functionality
    • Agent communication patterns
    • Result aggregation

Priority: HIGH - AI orchestration core

Data Service (sentinel_backend/data_service/)

Files to Test: main.py, models.py, schemas.py

Test Categories:

  • ✅ Data Models

    • Model validation and constraints
    • Relationship integrity
    • Data transformation logic

  • ✅ Analytics Functions

    • Test result aggregation
    • Performance metrics calculation
    • Report generation logic

Priority: MEDIUM - Supporting functionality

Execution Service (sentinel_backend/execution_service/)

Files to Test: main.py

Test Categories:

  • ✅ Test Execution Logic

    • Test case execution workflows
    • Result collection and storage
    • Error handling and recovery

  • ✅ Environment Management

    • Target environment validation
    • Configuration management
    • Resource cleanup

Priority: HIGH - Test execution core

2. Integration Tests

Service-to-Service Integration

  • ✅ API Gateway ↔ All Services

    • Request routing accuracy
    • Authentication token propagation
    • Error response handling

  • ✅ Orchestration ↔ Rust Core

    • Message passing via RabbitMQ
    • Task completion workflows
    • Agent result processing

  • ✅ Data Service ↔ Database

    • Transaction handling
    • Connection pooling
    • Query optimization

Database Integration

  • ✅ Schema Validation

    • Migration scripts testing
    • Data integrity constraints
    • Performance with sample data

  • ✅ Connection Handling

    • Connection pooling behavior
    • Failover scenarios
    • Connection recovery

External Dependencies

  • ✅ Message Broker (RabbitMQ)

    • Queue management
    • Message persistence
    • Consumer group behavior

  • ✅ Observability Stack

    • Prometheus metrics collection
    • Jaeger tracing functionality
    • Log aggregation

3. Performance Tests

  • ✅ Load Testing

    • Concurrent user simulation
    • API endpoint performance
    • Database query performance

  • ✅ Memory and Resource Usage

    • Memory leak detection
    • CPU usage under load
    • Database connection limits

Frontend Testing Plan

1. Component Unit Tests

Pages (sentinel_frontend/src/pages/)

Components to Test: Dashboard.js, Login.js, Specifications.js, TestCases.js, TestRuns.js, TestRunDetail.js, Analytics.js

Test Categories:

  • ✅ Rendering Tests

    • Component renders without crashing
    • Correct initial state display
    • Conditional rendering logic

  • ✅ User Interactions

    • Button clicks and form submissions
    • Navigation behavior
    • Input validation

  • ✅ Data Display

    • Props handling and display
    • Loading states
    • Error state handling

Components (sentinel_frontend/src/components/)

Components to Test: Layout.js, PrivateRoute.js

Test Categories:

  • ✅ Layout Component

    • Navigation menu functionality
    • User authentication state display
    • Responsive behavior

  • ✅ PrivateRoute Component

    • Authentication checks
    • Redirect behavior
    • Route protection logic

Redux Slices (sentinel_frontend/src/features/)

Slices to Test: authSlice.js, specificationsSlice.js

Test Categories:

  • ✅ State Management

    • Initial state correctness
    • Action creators functionality
    • Reducer logic validation

  • ✅ Async Thunks

    • API call handling
    • Loading state management
    • Error handling

Services (sentinel_frontend/src/services/)

Services to Test: api.js

Test Categories:

  • ✅ API Client
    • HTTP request methods
    • Authentication header handling
    • Response transformation
    • Error handling and retries

2. Integration Tests (Frontend)

User Workflows

  • ✅ Authentication Flow

    • Login process end-to-end
    • Token storage and retrieval
    • Logout functionality

  • ✅ Specification Management

    • Upload specification workflow
    • View and edit specifications
    • Delete specifications

  • ✅ Test Management

    • Create and run test suites
    • View test results
    • Export test data

API Integration

  • ✅ Backend API Integration
    • All API endpoints integration
    • Error handling for API failures
    • Loading states during API calls

End-to-End Testing Plan

1. Critical User Journeys

Complete Testing Workflow

  • ✅ Full Platform Workflow
    • User login → Upload spec → Generate tests → Run tests → View results
    • Multi-user scenarios
    • Permission-based access testing

Authentication & Authorization

  • ✅ RBAC Testing
    • Admin, Manager, Tester, Viewer role workflows
    • Permission enforcement across UI
    • Unauthorized access prevention

Data Persistence

  • ✅ Data Flow Testing
    • Specification upload to test execution
    • Result storage and retrieval
    • Data consistency across services

2. Cross-Browser & Device Testing

  • ✅ Browser Compatibility
    • Chrome, Firefox, Safari, Edge
    • Mobile responsive design
    • Performance across browsers

Docker-Based Test Infrastructure

1. Container Architecture for Testing

Existing Docker Test Setup

✅ Current Infrastructure:

  • docker-compose.test.yml - Complete test environment
  • Dockerfile.test - Dedicated test container with all dependencies
  • Isolated test network (sentinel_test_network)
  • Dedicated test database (test_db)
  • All services configured for testing environment

Test Execution Containers

Backend Test Container (sentinel_backend/Dockerfile.test):

FROM python:3.10-slim
# Includes: pytest, pytest-asyncio, pytest-cov, pytest-mock, 
#          pytest-xdist, httpx, factory-boy, freezegun

Frontend Test Container (To be created):

FROM node:18-alpine
# Will include: Jest, React Testing Library, Playwright, Coverage tools

Test Database Container

  • Image: postgres:15
  • Isolated: Separate from production DB
  • Health checks: Ensures DB ready before tests
  • Port: 15432 (isolated from main DB)

2. Docker Test Execution Strategies

All Tests in Containers (Recommended Approach)

Backend Tests:

# Run all backend tests in Docker
cd sentinel_backend
./run_tests.sh -d                    # Full test suite in Docker
./run_tests.sh -d -t unit            # Unit tests only
./run_tests.sh -d -t integration     # Integration tests
./run_tests.sh -d -t performance     # Performance tests

# Direct Docker Compose
docker-compose -f docker-compose.test.yml run --rm test_runner

Frontend Tests in Docker:

# Frontend tests in Docker (new capability)
docker-compose -f docker-compose.test.yml run --rm frontend_test_runner
docker-compose -f docker-compose.test.yml run --rm e2e_test_runner

Complete Test Suite:

# Run ALL tests (backend + frontend + E2E) in Docker
docker-compose -f docker-compose.test.yml up --abort-on-container-exit

3. Enhanced Docker Test Configuration

Updated docker-compose.test.yml (Additions needed)

Frontend Test Runner:

frontend_test_runner:
  build:
    context: ../sentinel_frontend
    dockerfile: Dockerfile.test
  container_name: sentinel_frontend_test_runner
  environment:
    - NODE_ENV=test
    - REACT_APP_API_URL=http://test_api_gateway:8000
  volumes:
    - ../sentinel_frontend:/app
    - frontend_test_reports:/app/coverage
  command: ["npm", "run", "test:coverage"]
  networks:
    - sentinel_test_network

E2E Test Runner:

e2e_test_runner:
  build:
    context: ../sentinel_frontend
    dockerfile: Dockerfile.e2e
  container_name: sentinel_e2e_test_runner
  environment:
    - NODE_ENV=test
    - REACT_APP_API_URL=http://test_api_gateway:8000
  volumes:
    - ../sentinel_frontend:/app
    - e2e_test_reports:/app/test-results
  depends_on:
    test_api_gateway:
      condition: service_healthy
    frontend_test_runner:
      condition: service_completed_successfully
  command: ["npx", "playwright", "test"]
  networks:
    - sentinel_test_network

4. Testing Tools & Frameworks (Containerized)

Backend (Already Configured)

  • Unit Tests: pytest with coverage
  • Integration Tests: pytest with Docker services
  • Performance Tests: pytest-benchmark
  • Mocking: pytest-mock, httpx-mock
  • Database: SQLAlchemy test fixtures
  • Async Testing: pytest-asyncio
  • Parallel Execution: pytest-xdist

Frontend (To be Added)

  • Unit Tests: Jest + React Testing Library
  • Component Tests: @testing-library/react
  • E2E Tests: Playwright (containerized)
  • Coverage: Built-in Jest coverage
  • Visual Testing: Playwright visual comparisons

5. Test Data Management in Docker

Database Test Data

# Test fixtures service
test_data_loader:
  build:
    context: .
    dockerfile: Dockerfile.test
  container_name: sentinel_test_data_loader
  environment:
    - SENTINEL_ENVIRONMENT=testing
  depends_on:
    test_db:
      condition: service_healthy
  command: ["python", "-m", "scripts.load_test_data"]
  volumes:
    - ./tests/fixtures:/app/fixtures
  networks:
    - sentinel_test_network

API Mock Services (If needed)

mock_external_apis:
  image: mockserver/mockserver:latest
  container_name: sentinel_mock_server
  ports:
    - "1080:1080"
  environment:
    MOCKSERVER_INITIALIZATION_JSON_PATH: /config/mock-config.json
  volumes:
    - ./tests/mocks:/config
  networks:
    - sentinel_test_network

6. CI/CD Integration with Docker

Test Execution Strategy

  • Pre-commit: Unit tests in Docker (fast feedback)
  • PR Validation: Full Docker test suite
  • Staging: Complete Docker test environment
  • Production: Smoke tests via Docker

Docker Test Commands for CI/CD

# GitHub Actions / CI Pipeline
docker-compose -f docker-compose.test.yml up --build --abort-on-container-exit --exit-code-from test_runner

# Individual test suites
docker-compose -f docker-compose.test.yml run --rm test_runner pytest tests/unit/
docker-compose -f docker-compose.test.yml run --rm test_runner pytest tests/integration/
docker-compose -f docker-compose.test.yml run --rm frontend_test_runner npm run test:coverage
docker-compose -f docker-compose.test.yml run --rm e2e_test_runner npx playwright test

7. Test Reports and Artifacts

Volume Mounts for Test Results

volumes:
  test_reports:
    driver: local
  frontend_test_reports:
    driver: local
  e2e_test_reports:
    driver: local

Coverage Report Access

# Backend coverage reports
docker-compose -f docker-compose.test.yml run --rm test_runner
# Reports available at: ./test_reports/htmlcov/index.html

# Frontend coverage reports  
docker-compose -f docker-compose.test.yml run --rm frontend_test_runner
# Reports available at: ./sentinel_frontend/coverage/lcov-report/index.html

8. Docker Test Environment Benefits

Isolation & Consistency

  • ✅ Isolated test database and network
  • ✅ Consistent environment across developers
  • ✅ No dependency on local installations
  • ✅ Easy cleanup and reset

Scalability & Performance

  • ✅ Parallel test execution with pytest-xdist
  • ✅ Service-level test isolation
  • ✅ Resource-controlled test execution
  • ✅ Multi-stage testing pipeline

Development Experience

  • ✅ Simple test execution commands
  • ✅ No local environment setup required
  • ✅ Fast iteration with volume mounts
  • ✅ Complete integration testing capability

Implementation Roadmap

Phase 1: Foundation (Weeks 1-2)

  • ✅ Set up testing infrastructure and CI/CD
  • ✅ Implement unit tests for Authentication Service
  • ✅ Implement unit tests for API Gateway core functionality
  • ✅ Set up frontend testing environment with additional tools

Phase 2: Core Services (Weeks 3-4)

  • ✅ Complete unit tests for Specification Service
  • ✅ Complete unit tests for Orchestration Service
  • ✅ Implement integration tests for service-to-service communication
  • ✅ Add frontend component tests for critical pages

Phase 3: Data & Execution (Weeks 5-6)

  • ✅ Complete unit tests for Data Service and Execution Service
  • ✅ Implement database integration tests
  • ✅ Add Redux slice and API service tests
  • ✅ Set up E2E testing framework

Phase 4: Integration & E2E (Weeks 7-8)

  • ✅ Implement comprehensive integration tests
  • ✅ Create E2E tests for critical user workflows
  • ✅ Performance testing and optimization
  • ✅ Documentation and team training

Phase 5: Refinement (Week 9)

  • ✅ Test coverage analysis and gaps closure
  • ✅ Performance optimization based on test results
  • ✅ Final documentation and handover

Success Metrics

Quantitative Metrics

  • Backend Code Coverage: ≥85%
  • Frontend Code Coverage: ≥80%
  • Integration Test Coverage: 100% of service interfaces
  • E2E Test Coverage: 100% of critical user paths
  • Test Execution Time: <5 minutes for unit tests, <15 minutes for full suite

Qualitative Metrics

  • Bug Detection: Increased early bug detection in development
  • Developer Confidence: Reduced fear of refactoring and changes
  • Deployment Safety: Safer releases with comprehensive test validation
  • Maintenance: Easier identification of regression issues

Progress Tracking

Implementation Status

  • Phase 1: Foundation - ✅ COMPLETED
    • Testing infrastructure set up
    • Auth Service fully tested
    • API Gateway fully tested
  • Phase 2: Core Services - ✅ COMPLETED
    • Spec Service fully tested
    • Orchestration Service fully tested
    • Service-to-service integration tests
  • Phase 3: Data & Execution - ✅ COMPLETED
    • Data Service fully tested
    • Execution Service fully tested
    • Factory pattern applied to all services
  • Phase 4: Integration & E2E - In Progress
    • Comprehensive integration tests needed
    • E2E tests for critical workflows pending
  • Phase 5: Refinement - Not Started
    • Performance optimization pending
    • Final documentation needed

Test Files to be Created

Backend Test Files (Current Status)

sentinel_backend/tests/
├── unit/ ✅ COMPLETED
│   ├── test_api_gateway.py ✅
│   ├── test_auth_service.py ✅
│   ├── test_auth_middleware.py ✅
│   ├── test_spec_service.py ✅
│   ├── test_orchestration_service.py ✅
│   ├── test_data_service.py ✅
│   ├── test_execution_service.py ✅
│   └── agents/
│       ├── test_functional_agents.py (pending)
│       ├── test_security_agents.py (pending)
│       └── test_performance_agents.py (pending)
├── integration/ ✅ PARTIALLY COMPLETED
│   ├── test_auth_integration.py ✅
│   ├── test_service_communication.py (pending)
│   ├── test_database_operations.py (pending)
│   ├── test_message_broker.py (pending)
│   └── test_rust_core_integration.py ✅
├── performance/ (pending)
│   ├── test_load_performance.py
│   └── test_memory_usage.py
├── fixtures/ ✅ COMPLETED
│   ├── auth_fixtures.py ✅
│   ├── spec_fixtures.py ✅
│   └── mock_responses.py ✅
└── helpers/ ✅ COMPLETED
    └── auth_helpers.py ✅

Frontend Test Files

sentinel_frontend/src/
├── pages/
│   ├── Login.test.js
│   ├── Specifications.test.js
│   ├── TestCases.test.js
│   ├── TestRuns.test.js
│   ├── TestRunDetail.test.js
│   └── Analytics.test.js
├── components/
│   ├── Layout.test.js
│   └── PrivateRoute.test.js
├── features/
│   ├── auth/
│   │   └── authSlice.test.js
│   └── specifications/
│       └── specificationsSlice.test.js
├── services/
│   └── api.test.js
└── __tests__/
    ├── integration/
    │   ├── authFlow.test.js
    │   ├── specificationFlow.test.js
    │   └── testExecutionFlow.test.js
    └── e2e/
        ├── completeWorkflow.test.js
        ├── userRoles.test.js
        └── crossBrowser.test.js

Quick Start Guide - Running Tests in Docker

Simple Commands (Recommended)

From the project root directory:

# Run complete test suite (backend + frontend + E2E)
./run-all-tests.sh

# Run specific test types
./run-all-tests.sh --backend-only        # Backend tests only
./run-all-tests.sh --frontend-only       # Frontend tests only  
./run-all-tests.sh --e2e-only            # E2E tests only

# Run specific test categories
./run-all-tests.sh -t unit               # Unit tests only
./run-all-tests.sh -t integration        # Integration tests only
./run-all-tests.sh -t performance        # Performance tests only

# With additional options
./run-all-tests.sh --verbose --no-cleanup   # Verbose output, keep containers

Advanced Commands

From the sentinel_backend directory:

# Backend-specific commands
./run_tests.sh -d                        # All backend tests in Docker
./run_tests.sh -d -t unit                # Unit tests in Docker
./run_tests.sh -d -t integration         # Integration tests in Docker

# Frontend-specific commands  
./run_tests.sh -d -f                     # Frontend tests in Docker
./run_tests.sh -d -t e2e                 # E2E tests in Docker

# Complete test suite
./run_tests.sh -d                        # Everything in Docker

Direct Docker Compose Commands

cd sentinel_backend

# Individual test runners
docker-compose -f docker-compose.test.yml run --rm test_runner
docker-compose -f docker-compose.test.yml run --rm frontend_test_runner
docker-compose -f docker-compose.test.yml run --rm e2e_test_runner

# Start full test environment
docker-compose -f docker-compose.test.yml up --abort-on-container-exit

# Cleanup
docker-compose -f docker-compose.test.yml down -v

Accessing Test Reports

After running tests, coverage reports are available at:

  • Backend Coverage: sentinel_backend/test_reports/htmlcov/index.html
  • Frontend Coverage: sentinel_frontend/coverage/lcov-report/index.html
  • E2E Results: sentinel_frontend/test-results/
  • Playwright Report: sentinel_frontend/playwright-report/index.html

Development Workflow

  1. Development: Make changes to code
  2. Quick Check: ./run-all-tests.sh -t unit (fast feedback)
  3. Pre-commit: ./run-all-tests.sh --backend-only (validate backend changes)
  4. Pre-PR: ./run-all-tests.sh (complete test suite)
  5. Review Reports: Open coverage reports in browser

Troubleshooting

Docker Issues:

# Clean up Docker resources
docker system prune -f
docker volume prune -f

# Rebuild test containers
docker-compose -f sentinel_backend/docker-compose.test.yml build --no-cache

Permission Issues:

# Make scripts executable
chmod +x ./run-all-tests.sh
chmod +x ./sentinel_backend/run_tests.sh

Notes

  • This plan prioritizes critical path functionality first (Authentication, API Gateway, Core Services)
  • Test implementation should follow TDD principles where possible
  • Regular test coverage reports should be generated and reviewed
  • Performance benchmarks should be established and monitored
  • Tests should be deterministic and not flaky
  • All test data should be properly isolated and cleaned up

Last Updated: August 2025 Plan Status: Phase 3 Completed - Factory Pattern Implementation Complete