Skip to content

test: improve pytest coverage and test infrastructure#3339

Draft
deacon-mp wants to merge 3 commits intomasterfrom
pytest
Draft

test: improve pytest coverage and test infrastructure#3339
deacon-mp wants to merge 3 commits intomasterfrom
pytest

Conversation

@deacon-mp
Copy link
Contributor

Summary

Work-in-progress branch improving test coverage and test infrastructure:

  • Updated pytest.ini configuration
  • Improved test coverage across API handlers
  • Added mcp and range plugin submodules
  • Updated package.json test dependencies

Status

Draft PR - needs cleanup and review before merge.

@deacon-mp deacon-mp requested a review from Copilot March 16, 2026 04:07
@gitguardian
Copy link

gitguardian bot commented Mar 16, 2026

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
- - Generic High Entropy Secret 6c7f351 conf/default.yml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves the project’s pytest setup and expands/modernizes async test infrastructure while adding/adjusting config and plugin dependencies needed to run the updated tests.

Changes:

  • Migrates many fixtures/tests from event_loop.run_until_complete(...) to native async/await and pytest-asyncio auto mode.
  • Refactors several tests (DNS contact, API v2 handlers) to reduce duplication and align with updated service initialization patterns.
  • Adds new plugin submodules and updates runtime configs/dependencies to support the new test paths.

Reviewed changes

Copilot reviewed 27 out of 29 changed files in this pull request and generated 13 comments.

Show a summary per file
File Description
tests/services/test_rest_svc.py Updates expected REST operation payload value types.
tests/contacts/test_contact_dns.py Large refactor of DNS contact tests/fixtures and adds file-upload test polling.
tests/conftest.py Reworks core pytest fixtures/service initialization and introduces async fixtures.
tests/api/v2/test_security.py Converts webapp fixture/service setup to async/await.
tests/api/v2/test_responses.py Adjusts webapp fixture signature (removes event_loop dependency).
tests/api/v2/test_knowledge.py Refactors knowledge API tests and adds an operation-based fact test.
tests/api/v2/managers/test_base_api_manager.py Switches manager tests to use StubDataService rather than real data_svc.
tests/api/v2/handlers/test_sources_api.py Converts source fixture to async store.
tests/api/v2/handlers/test_schedules_api.py Converts multiple schedule fixtures to async and updates operation payload usage.
tests/api/v2/handlers/test_plugins_api.py Converts plugin fixtures to async store/dump.
tests/api/v2/handlers/test_planners_api.py Converts planner fixtures to async store and updates payload fixture.
tests/api/v2/handlers/test_operations_api.py Updates operation assertions to use schema-loaded objects and Link objects.
tests/api/v2/handlers/test_objectives_api.py Converts objective fixtures to async store.
tests/api/v2/handlers/test_obfuscators_api.py Converts obfuscator fixture to async store.
tests/api/v2/handlers/test_contacts_api.py Makes contact report fixture async for service readiness.
tests/api/v2/handlers/test_agents_api.py Converts agent/ability fixtures to async store.
tests/api/v2/handlers/test_adversaries_api.py Converts adversary fixture to async store and relaxes one equality assertion.
tests/api/v2/handlers/test_abilities_api.py Converts ability fixture to async store and relaxes response equality assertions.
test.txt Adds a captured pytest run output (currently includes a failing test).
pytest.ini Enables pytest-asyncio auto mode.
plugins/range Adds range plugin as a git submodule reference.
plugins/mcp Adds mcp plugin as a git submodule reference.
package.json Adds JS dependencies used by tests/utilities.
data/planners/123.yml Adds a planner fixture file for tests/config.
conf/payloads.yml Populates payload configuration entries (previously empty).
conf/default.yml Updates default config keys, plugins, and contact settings.
conf/agents.yml Reorders/expands deployment IDs in agent config.
.gitignore Adds ignore rules for plugins/* (duplicated).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

'adversary': {'description': 'an empty adversary profile', 'name': 'ad-hoc',
'adversary_id': 'ad-hoc', 'atomic_ordering': [],
'objective': '495a9828-cab1-44dd-a0ca-66e58177d8cc',
'objective': uuid.UUID('495a9828-cab1-44dd-a0ca-66e58177d8cc'),
Comment on lines +280 to +284
max_wait = 5 # seconds
start = time.time()
while not os.path.isfile(target_path) and time.time() - start < max_wait:
time.sleep(0.1)
assert os.path.isfile(target_path)
Comment on lines +264 to +288
for idx, qname in enumerate(get_file_upload_metadata_qnames(message_id, metadata_chunks)):
response = await get_dns_response(qname, 'a')
assert_successful_ipv4(response)
if idx == len(metadata_chunks) - 1:
assert_odd_ipv4(response)
else:
assert_even_ipv4(response)

for idx, qname in enumerate(get_file_upload_data_qnames(message_id, file_chunks)):
response = await get_dns_response(qname, 'a')
assert_successful_ipv4(response)
if idx == len(file_chunks) - 1:
assert_odd_ipv4(response)
else:
assert_even_ipv4(response)

max_wait = 5 # seconds
start = time.time()
while not os.path.isfile(target_path) and time.time() - start < max_wait:
time.sleep(0.1)
assert os.path.isfile(target_path)
decrypted = get_decrypted_upload(target_path)
assert decrypted == file_data
os.remove(target_path)
os.rmdir(target_dir)
BaseWorld.apply_config('main', yaml.load(c, Loader=yaml.FullLoader))
BaseWorld.apply_config('agents', BaseWorld.strip_yml(os.path.join(CONFIG_DIR, 'agents.yml'))[0])
BaseWorld.apply_config('payloads', BaseWorld.strip_yml(os.path.join(CONFIG_DIR, 'payloads.yml'))[0])
pytest_plugins = 'pytest_asyncio'
Comment on lines +195 to +196
async def fact():
async def _generate_fact(trait, *args, **kwargs):
Comment on lines +358 to +360
f.trait == response[0]['trait'] and
f.value == response[0]['value'] and
f.source == response[0]['source']
assert result['paw'] == payload['paw']
assert result['id']
assert result['ability']['name'] == 'Manual Command'
assert 'ability' in result and result['ability']['name']
Comment on lines +1 to +38
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0
rootdir: /home/caldera/Desktop/CalderaVENV/caldera
configfile: pytest.ini
plugins: anyio-4.9.0, aiohttp-1.1.0, asyncio-1.0.0
asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 10 items

tests/contacts/test_contact_dns.py ........F. [100%]

=================================== FAILURES ===================================
_______________________________ test_file_upload _______________________________
tests/contacts/test_contact_dns.py:284: in test_file_upload
assert os.path.isfile(target_path)
E AssertionError: assert False
E + where False = <function isfile at 0x73828cd18360>('/tmp/testhost-asdasd/testupload.txt-92811837')
E + where <function isfile at 0x73828cd18360> = <module 'posixpath' (frozen)>.isfile
E + where <module 'posixpath' (frozen)> = os.path
------------------------------ Captured log setup ------------------------------
ERROR asyncio:base_events.py:1821 Task was destroyed but it is pending!
task: <Task pending name='Task-246' coro=<Contact.operation_loop() done, defined at /home/caldera/Desktop/CalderaVENV/caldera/app/contacts/contact_tcp.py:27> wait_for=<Future pending cb=[Task.task_wakeup()]>>
=============================== warnings summary ===============================
../lib/python3.12/site-packages/pyasn1/codec/ber/encoder.py:952
/home/caldera/Desktop/CalderaVENV/lib/python3.12/site-packages/pyasn1/codec/ber/encoder.py:952: DeprecationWarning: tagMap is deprecated. Please use TAG_MAP instead.
warnings.warn(f"{attr} is deprecated. Please use {newAttr} instead.", DeprecationWarning)

../lib/python3.12/site-packages/pyasn1/codec/ber/encoder.py:952
/home/caldera/Desktop/CalderaVENV/lib/python3.12/site-packages/pyasn1/codec/ber/encoder.py:952: DeprecationWarning: typeMap is deprecated. Please use TYPE_MAP instead.
warnings.warn(f"{attr} is deprecated. Please use {newAttr} instead.", DeprecationWarning)

tests/contacts/test_contact_dns.py::test_config
/home/caldera/Desktop/CalderaVENV/caldera/app/service/app_svc.py:36: DeprecationWarning: There is no current event loop
self.loop = asyncio.get_event_loop()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/contacts/test_contact_dns.py::test_file_upload - AssertionError:...
=================== 1 failed, 9 passed, 3 warnings in 16.57s ===================
api_key_blue: BLUEADMIN123
api_key_red: ADMIN123
api_key_blue: ADMIN123
api_key_red: GMQfwZSX_KlBTnxy4eM3RneD2Hw4xuxoypWzALkVvK8
encryption_key: ADMIN123
auth.login.handler.module: default
crypt_salt: 6-RzX6yUK6jWinrWMQ4eifneLTYOCPA_ibBMObK_XXg
encryption_key: OGya-CCywKXCahDkCz9RBEacBMhDc4NkBp8lRvgZz28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants