Skip to content

CLAUDE.md

Latest commit

 

History

History
108 lines (83 loc) · 6.81 KB

CLAUDE.md

File metadata and controls

108 lines (83 loc) · 6.81 KB
Raw

Agentic-TAF

Approach

  • Think before acting. Read existing files before writing code.
  • Be concise in output but thorough in reasoning.
  • Prefer editing over rewriting whole files.
  • Do not re-read files already read unless they may have changed.
  • Test code before declaring done. Run relevant lint/typecheck/test after every change.
  • No sycophantic openers or closing fluff.
  • Keep solutions simple and direct.
  • User instructions always override this file.

Project

Agentic Test Automation Framework — an extensible, plugin-based, multi-layered Python framework for test automation across API, Web UI, WebSocket, CLI, AI/LLM validation, and chaos engineering. Evolved from uiXautomation (PyXTaf), modernized for Python 3.12+.

Part of the Agentic QA Platform (6 repos). Design authority and implementation plans live in agentic-qa-platform. This repo contains the framework core and platform test suites (eat your own dogfood).

Language: Python 3.12+

Architecture

Four-layer plugin architecture (top to bottom):

  1. Test Suites (src/test/python/) — ut/ (293 unit tests), suites/agentic/ (63 E2E + 10 BDD: 21 API + 8 security + 10 UI + 16 AI (test_ai 11 + test_e2e_quality 5) + 4 chaos + 4 load; 10 BDD across 4 feature files via behave), bpt/ (BDD/ATDD examples). suites/agentic/reporting/ is a CI utility (JUnit to OpenSearch push), not a test suite. The llm_judge fixture lives in shared agentic/conftest.py; non-AI suites can opt in via llm_judge_optional (returns None if unavailable) or the chat_and_judge composite fixture.
  2. Modeling (src/main/python/taf/modeling/) — Browser, RESTClient, CLIRunner, WSClient, LLMJudge, ChaosRunner
  3. Foundation (src/main/python/taf/foundation/) — ServiceLocator, Configuration (YAML), Utils
  4. Plugins (src/main/python/taf/foundation/plugins/) — Concrete implementations discovered at runtime via ServiceLocator

Plugin interfaces in taf/foundation/api/plugins/:

Interface Implementation Status
WebPlugin SeleniumPlugin (default, headless) Implemented
WebPlugin PlaywrightPlugin (optional) Implemented
RESTPlugin RequestsPlugin (default) Implemented
RESTPlugin HttpxRESTPlugin (optional) Implemented
WSPlugin WebSocketPlugin (optional) Implemented
CLIPlugin ParamikoPlugin Implemented
MobilePlugin AppiumPlugin Stub / planned (interface defined; concrete plugin not yet implemented)
LLMPlugin LLMJudgePlugin (optional, OpenAI/Anthropic) Implemented
ChaosPlugin K8sChaosPlugin (optional) Implemented

Build & Test

# Lint
flake8 src/ --max-line-length=120

# Type check
mypy src/main/python/taf/ --ignore-missing-imports

# Framework unit tests (293 tests)
PYTHONPATH=src/main/python pytest src/test/python/ut/ -v

Conventions

  • Files/dirs: lowercase-with-hyphens for directories, lowercase_with_underscores for Python modules.
  • Plugins: One plugin class per file. File named <name>plugin.py (e.g., playwrightplugin.py).
  • Plugin interfaces: Abstract base in taf/foundation/api/plugins/. Concrete in taf/foundation/plugins/<type>/<name>/.
  • Modeling: High-level wrappers in taf/modeling/<type>/. Must use ServiceLocator to resolve plugins, never import concrete plugins directly.
  • Config: YAML files in taf/foundation/conf/ (framework). Environment variables override YAML values (TAF_PLUGIN_<NAME>_<KEY>, TAF_LLM_PROVIDER, TAF_LLM_MODEL, TAF_LLM_BASE_URL, TAF_LLM_API_KEY).
  • Commits: <scope>: <description> — scopes: framework, plugin, modeling, test, ci, docs.
  • Copyright: Copyright (c) 2017-2026 Wesley Peng — Apache-2.0 license (relicensed from LGPL-3.0 in 2026; every file carries an SPDX header).
  • No secrets: Never hardcode credentials, IPs, or tokens. Use config files or env vars.

Key Patterns

ServiceLocator (plugin discovery)

from taf.foundation import ServiceLocator
from taf.foundation.api.plugins import WebPlugin, RESTPlugin

# Resolves concrete plugin based on config.yml
Browser = ServiceLocator.get_app_under_test(WebPlugin)
client = ServiceLocator.get_client(RESTPlugin)

Adding a new plugin

  1. Create interface in taf/foundation/api/plugins/<name>plugin.py (extend BasePlugin metaclass)
  2. Create base client in taf/foundation/api/<type>/client.py
  3. Create concrete implementation in taf/foundation/plugins/<type>/<name>/
  4. Register in taf/foundation/conf/config.yml
  5. Add modeling wrapper in taf/modeling/<type>/ if needed
  6. Write unit test in src/test/python/ut/

Pitfalls

  • Never import concrete plugins in test code — always go through ServiceLocator or modeling layer. Direct imports break plugin swappability.
  • ServiceLocator is a singleton — plugin resolution happens once per type. Configuration must be set before first access.
  • config.yml location is relative — plugin paths are relative to taf/foundation/conf/. Use ../plugins/... pattern.
  • Selenium 4 API — use find_elements(By.ID, value) not deprecated find_elements_by_id(). Use Service and Options, not executable_path or desired_capabilities.
  • LLM provider selection — default is openai (OpenAI-compatible). Set TAF_LLM_PROVIDER=anthropic or pass provider='anthropic' for native Anthropic API. TAF_LLM_MODEL, TAF_LLM_BASE_URL, TAF_LLM_API_KEY fallbacks let LLMJudge() with no args self-configure (used by the agentic conftest's session-scoped fixture).
  • LLMJudge must extend LLMClient, NOT Client — Client is the abstract base where evaluate()/score() raise NotImplementedError. LLMJudge(LLMClient) inherits the working implementations + provider registry. The original LLMJudge(Client) silently broke at first real use because Client.evaluate() raises. Closed in db10c46.
  • LLMJudge() eagerly builds a ChatOpenAI — instantiating bare LLMJudge() calls LLMClient.__init__ which calls _create_chat_model(...) and validates the API key. Tests that build LLMJudge() must unittest.mock.patch('taf.foundation.plugins.llm.judge.llmclient._create_chat_model', return_value=MagicMock()) in setUp.
  • Optional plugins — websocket, llm, chaos plugins are enabled: False by default. Install the optional dep (pip install .[chaos]) and set enabled: True or use env var override.
  • Configuration env overrides are case-insensitive — TAF_PLUGIN_REST_NAME matches config key REST (uppercase). The lookup normalizes to lowercase.
  • E2E tests use ServiceLocator — never import httpx.Client or concrete plugins directly. Use conftest.py fixtures that resolve via ServiceLocator.get_client(RESTPlugin) with env override.
  • BDD tests use behave, not pytest-bdd — existing examples in src/test/python/bpt/bdd/ use behave with Gherkin.