Agentic Test Automation Framework — an extensible, plugin-based, multi-layered framework for test automation across API, Web UI, WebSocket, CLI, and AI/LLM validation.
Evolved from PyXTaf (uiXautomation), modernized for Python 3.12+.
Original PyXTaf architecture (v0.x)
┌─────────────────────────────────────────────────────────────────────────┐
│ Test Suites (pytest / behave) │
│ Unit tests (ut/) │ BDD/ATDD examples (bpt/) │ Platform (agentic/) │
├─────────────────────────────────────────────────────────────────────────┤
│ Modeling Layer │
│ RESTClient │ Browser │ CLIRunner │ WSClient │ LLMJudge │ ChaosRunner │
├─────────────────────────────────────────────────────────────────────────┤
│ Plugin Layer │
│ Selenium │ Playwright │ Requests │ httpx │ WS │ Paramiko │ LLM │Chaos │
├─────────────────────────────────────────────────────────────────────────┤
│ Foundation Layer │
│ ServiceLocator │ Configuration (YAML) │ Utils │
└─────────────────────────────────────────────────────────────────────────┘
The framework uses a ServiceLocator pattern with pluggable backends. Each plugin type defines an interface; concrete implementations are discovered at runtime via YAML configuration.
Implemented:
| Plugin Interface | Implementation | Purpose |
|---|---|---|
WebPlugin |
SeleniumPlugin (default) |
Browser automation (Selenium 4, headless) |
WebPlugin |
PlaywrightPlugin (optional) |
Browser automation (Playwright, headless) |
RESTPlugin |
RequestsPlugin (default) |
REST API testing (requests) |
RESTPlugin |
HttpxRESTPlugin (optional) |
REST API testing (httpx) |
WSPlugin |
WebSocketPlugin (optional) |
WebSocket streaming (websockets) |
CLIPlugin |
ParamikoPlugin |
SSH / CLI access |
MobilePlugin |
AppiumPlugin (stub / planned) |
Mobile automation — interface defined, concrete plugin not yet implemented |
LLMPlugin |
LLMJudgePlugin (optional) |
LLM response quality evaluation (OpenAI/Anthropic) |
ChaosPlugin |
K8sChaosPlugin (optional) |
K8s chaos engineering (pod kill, network partition, Flux suspend) |
Optional plugins require their dependency installed (
pip install agentic-taf[chaos], etc.) and a config.yml change to enable. Defaults remain Selenium + Requests.
Foundation (taf/foundation/)
ServiceLocator— Plugin discovery and dependency injection via metaclass-based registryConfiguration— YAML-based config with environment variable overrides (TAF_PLUGIN_<NAME>_<KEY>)BasePlugin— Metaclass that auto-registers plugin implementationsUtils— Logger, YAML data model, connection cache, serialization traits
Modeling (taf/modeling/)
- High-level abstractions that compose plugin capabilities into test-friendly APIs
Browser— Page navigation, screenshot, element interaction (wraps WebPlugin)RESTClient— HTTP client with JSON encode/decode (wraps RESTPlugin)CLIRunner— SSH command execution (wraps CLIPlugin)WSClient— WebSocket streaming withcollect(),collect_text(),send_and_receive()LLMJudge— LLM-as-judge withassert_quality()threshold assertions (OpenAI/Anthropic)ChaosRunner— Chaos experiment lifecycle withassert_resilient()retry/timeout
Test Suites (src/test/python/)
ut/— 274 framework unit tests (all pass)suites/agentic/api/— 21 E2E API tests (contract, functional, state machine)suites/agentic/security/— 8 E2E security tests (RBAC, secret exposure, injection)suites/agentic/ui/— 10 E2E UI tests (Playwright, engine-agnostic Page Objects)suites/agentic/ai/— 16 E2E AI tests: 11 baseline (test_ai.py) + 5 ground-truth & multi-turn (test_e2e_quality.py) — LLM-as-judge evaluation, adversarial, fallback; skip if LLM downsuites/agentic/chaos/— 4 chaos experiments (K8sChaosPlugin: pod kill, Flux suspend, concurrent)suites/agentic/load/— 4 load tests (API throughput, WebSocket scale, provision throughput, chat latency)suites/agentic/bdd/— 10 BDD scenarios via behave across 4 feature files (provisioning, chat, LLM routing, environment lifecycle) — separate from pytest E2E countsuites/agentic/reporting/— CI utility module (JUnit to OpenSearch push, not a test suite)bpt/— BDD/ATDD examples (Bing search, httpbin API)- Totals: 274 unit + 63 E2E (pytest) + 10 BDD (behave)
agentic-taf/
├── src/
│ ├── main/python/taf/ # Framework core
│ │ ├── foundation/
│ │ │ ├── api/
│ │ │ │ ├── plugins/ # Plugin interfaces (8 types)
│ │ │ │ ├── ui/ # UI element abstractions
│ │ │ │ ├── svc/REST/ # REST client base class
│ │ │ │ ├── cli/ # CLI client base class
│ │ │ │ ├── ws/ # WebSocket client base class
│ │ │ │ ├── llm/ # LLM client base class
│ │ │ │ └── chaos/ # Chaos client base class
│ │ │ ├── plugins/ # Concrete implementations (8 implemented + 1 stub)
│ │ │ │ ├── web/selenium/ # SeleniumPlugin (default)
│ │ │ │ ├── web/playwright/ # PlaywrightPlugin (optional)
│ │ │ │ ├── svc/requests/ # RequestsPlugin (default)
│ │ │ │ ├── svc/httpx/ # HttpxRESTPlugin (optional)
│ │ │ │ ├── ws/websocket/ # WebSocketPlugin (optional)
│ │ │ │ ├── cli/paramiko/ # ParamikoPlugin
│ │ │ │ ├── mobile/appium/ # AppiumPlugin (stub / planned)
│ │ │ │ ├── llm/judge/ # LLMJudgePlugin (optional)
│ │ │ │ └── chaos/k8s/ # K8sChaosPlugin (optional)
│ │ │ ├── conf/ # YAML config + loader
│ │ │ ├── servicelocator.py # Plugin DI container
│ │ │ └── utils/ # Logger, YAMLData, traits
│ │ └── modeling/ # High-level test models (6 types)
│ │ ├── web/ # Browser + typed web controls
│ │ ├── svc/ # RESTClient
│ │ ├── cli/ # CLIRunner
│ │ ├── ws/ # WSClient
│ │ ├── llm/ # LLMJudge
│ │ └── chaos/ # ChaosRunner
│ │
│ └── test/python/
│ ├── ut/ # Framework unit tests (274 tests)
│ ├── suites/agentic/ # Platform E2E test suites
│ │ ├── api/ # API tests (21 tests)
│ │ ├── security/ # Security tests (8 tests)
│ │ ├── ui/ # UI tests (10 tests, Playwright)
│ │ │ └── pages/ # Page Objects (engine-agnostic)
│ │ ├── ai/ # AI tests (16 tests: test_ai 11 + test_e2e_quality 5)
│ │ ├── bdd/features/ # BDD scenarios (10 scenarios across 4 feature files, behave)
│ │ │ └── steps/ # Step definitions
│ │ ├── chaos/ # Chaos experiments (4 tests)
│ │ ├── load/ # Load & performance tests (4 tests)
│ │ ├── config/ # Environment configs
│ │ └── contract/schemas/ # OpenAPI schema
│ └── bpt/ # BDD/ATDD examples
│
├── Jenkinsfile # Jenkins CI pipeline (E2E stages gated by TAF_RUN_E2E)
├── pyproject.toml # Build config + dependencies
├── sonar-project.properties # SonarQube scanner config
├── .github/workflows/ci.yml # CI: lint → test → contract → build → docker
├── Dockerfile # Test runner container (Python 3.12 + Playwright)
├── docker-compose.yml # Local dev services
└── README.md
# From source (development)
pip install -r src/main/python/requirements-dev.txt
# Or via pyproject.toml extras
pip install -e ".[dev]"
# Run framework unit tests
PYTHONPATH=src/main/python pytest src/test/python/ut/ -vPlugins are configured via YAML and discovered by the ServiceLocator at runtime:
# taf/foundation/conf/config.yml
plugins:
web:
name: SeleniumPlugin
location: ../plugins/web/selenium
enabled: true
cli:
name: ParamikoPlugin
location: ../plugins/cli/paramiko
enabled: true
REST:
name: RequestsPlugin
location: ../plugins/svc/requests
enabled: true
mobile:
name: AppiumPlugin
location: ../plugins/mobile/appium
enabled: falseOverride via environment variables: TAF_PLUGIN_WEB_ENABLED=false
from taf.foundation import ServiceLocator
from taf.foundation.api.plugins import WebPlugin, RESTPlugin
# Get browser (resolves SeleniumPlugin based on config)
Browser = ServiceLocator.get_app_under_test(WebPlugin)
# Get REST client (resolves RequestsPlugin based on config)
client = ServiceLocator.get_client(RESTPlugin)from taf.modeling.web import Browser
from taf.modeling.web import WebButton, WebTextBox
browser = Browser(name='chrome', headless=True)
browser.launch('http://example.com')
txt_search = WebTextBox(id='search_box')
btn_go = WebButton(id='submit_btn')
txt_search.set('test query')
btn_go.click()This project was originally created as uiXautomation (PyXTaf) — a Python 2/3 compatible test automation framework with Selenium, Appium, Paramiko, and Requests plugins. It has been renamed to Agentic-TAF and modernized for Python 3.12+ with Selenium 4 support.
New plugin interfaces (Playwright, httpx, WebSocket, LLM-as-judge, K8s Chaos) and platform test suites (API, UI, AI, BDD, chaos, security, load — 63 E2E + 10 BDD) are implemented. AppiumPlugin is currently a stub (interface defined; concrete plugin planned for a future release). See docs/implementation-plan.md for the full roadmap.