docs: plan for #298 — custom test extension scaffolding by posit-vip-triage[bot] · Pull Request #299 · posit-dev/vip

posit-vip-triage · 2026-05-29T18:30:15Z

This plan outlines adding a vip scaffold subcommand and a comprehensive example showing how to extend VIP with custom tests for cross-product validation scenarios (verifying specific R/Python runtime versions across Workbench and Connect).

Scope: The scaffolded example focuses on runtime version verification only using ConnectClient.r_versions() / .python_versions() APIs. Package-install verification requires more complex patterns (bundle deployment and log scraping for Connect, session-exec for Workbench) and is out of scope for the initial scaffold. The example includes inline comments pointing users to src/vip_tests/connect/test_packages.py for deeper install-verification patterns.

The plan also addresses naming consistency across example directories and considers future extensibility for multi-template scaffolding support, including potential examples for SSL certificate validation, API performance testing, and user provisioning verification. The examples/custom_tests/ rename is deferred to a follow-up to keep this PR focused on the scaffolding feature.

Key requirements:

Scaffolded .feature files must include @connect @workbench auto-skip tags
conftest.py fixtures include sensible defaults (e.g., R 4.4.0/4.3.3, Python 3.12.0/3.11.0)
Selftests assert on scaffolded content (tags, imports, fixtures) not just file existence

Merging this PR will trigger an implementation PR — comment to iterate on the plan first.

> Generated by Triage open issues on posit-dev/vip for issue #298 · ● 7.9M · ◷

Add plan for a vip scaffold subcommand and comprehensive GxP validation example showing how to verify specific R/Python versions and packages across Workbench and Connect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-05-29T18:36:27Z

Preview Links

Website	Validation Report

https://posit-dev.github.io/vip/pr-preview-site/pr-299/	https://posit-dev.github.io/vip/pr-preview/pr-299/

ian-flores · 2026-05-29T18:42:02Z

+- **UNCONFIRMED:** Should the example test directory be named `examples/gxp_validation/` or something more generic like `examples/cross_product_validation/`?
+  - **Leaning toward:** `examples/cross_product_validation/` because the pattern applies beyond GxP (any customer who needs to lock runtime/package versions).


@copilot cross-product validation

Updated the plan to use examples/cross_product_validation/ throughout — all gxp_validation references replaced and the naming question marked as CONFIRMED. (commit: docs: confirm cross_product_validation naming, update all references)

ian-flores · 2026-05-29T18:42:20Z

+- **UNCONFIRMED:** Should the scaffold command default to copying from `examples/cross_product_validation/` or should it be extensible (multiple scaffold templates)?
+  - **Leaning toward:** single template for now; extensibility can be added later if demand emerges.


@copilot single for now

Confirmed — marked as single template for now with extensibility added later if needed.

ian-flores · 2026-05-29T18:43:24Z

+
+## Out of scope
+
+- Modifying the existing `examples/custom_tests/` directory (it serves a different purpose: minimal HTTP check). The new example is complementary, not a replacement.


@copilot it is weird to have one folder named custom_tests while another is named cross_product_validation. How many others example could we think of adding and if so we should add a way to support multiple.

Added new open question about renaming custom_tests → http_health_check for consistency. Listed potential future examples (SSL validation, API performance, user provisioning) and noted the Architecture section can support --template option for multi-template scaffolding.

Co-authored-by: ian-flores <18703558+ian-flores@users.noreply.github.com>

ian-flores · 2026-06-08T22:41:36Z

@copilot please revise this plan to address the following review feedback before it triggers the implementation agent:

1. (Blocker) Don't overpromise on package-install verification. The plan says the cross-product example will "verify R and Python packages install successfully on both products," but no client API supports this today. ConnectClient only exposes r_versions() / python_versions() (version listing); real install verification requires the deploy-a-bundle-and-scrape-logs flow in src/vip_tests/connect/test_packages.py. WorkbenchClient has no package surface at all — that path depends on the session-exec primitive tracked in #301 (plan PR #317). Either:

(preferred) scope the example down to version-only checks using connect_client.r_versions() / .python_versions(), with a comment pointing to connect/test_packages.py for the deeper install pattern; or
explicitly describe the bundle-deploy approach for Connect and acknowledge the Workbench gap / feat(workbench): add an in-session execution primitive #301 dependency.

2. Require @connect @workbench auto-skip tags. Add an explicit Components bullet stating the scaffolded .feature file must carry these tags, so tests skip (not fail) when only one product is configured — essential for a cross-product example run against partial deployments.

3. Resolve the examples/custom_tests/ rename. It's currently UNCONFIRMED. Either commit to it with a note to update the inline docstring path references (extension_dirs = ["/path/to/custom_tests"]) inside the moved files, or defer the rename entirely to a follow-up. Don't leave it half-decided.

4. Give the scaffolded conftest.py fixtures sensible defaults. required_r_packages / required_python_packages should have concrete defaults (or be wired to vip.toml) so the scaffold is immediately runnable, rather than leaving the implementation agent to guess.

5. Strengthen the selftest. Beyond file-existence, assert on scaffolded content (e.g. the .feature carries the auto-skip tags, the step file imports from pytest_bdd) to guard the templates against regression.

Also ensure the Verification section keeps examples/ in the ruff check/format paths (per CLAUDE.md) and includes just check.

@connect

- Scope down to runtime version checks only (no package install) - Document API limitations (ConnectClient.r/python_versions only) - Point to test_packages.py for deeper patterns - Require @connect @workbench auto-skip tags at feature level - Defer custom_tests rename to follow-up - Add sensible fixture defaults (R 4.4.0/4.3.3, Python 3.12.0/3.11.0) - Strengthen selftest with content assertions (tags, imports, fixtures) - Add just check to Verification section - Clarify Out of scope section Co-authored-by: ian-flores <18703558+ian-flores@users.noreply.github.com>

Copilot · 2026-06-08T22:45:44Z

@copilot please revise this plan to address the following review feedback before it triggers the implementation agent:

1. (Blocker) Don't overpromise on package-install verification. The plan say...

All review feedback addressed:

1. Blocker resolved — Scoped to runtime version checks only using ConnectClient.r_versions() / .python_versions(). Documented API limitations and pointed to src/vip_tests/connect/test_packages.py for install-verification patterns. Workbench gap acknowledged (blocked on #301).

2. Auto-skip tags — Required @connect @workbench tags at feature level in Components and Architecture sections.

3. Rename resolved — Deferred custom_tests → http_health_check rename to follow-up (marked as DEFERRED with rationale about inline path references).

4. Sensible defaults — Added concrete fixture examples: required_r_versions = ["4.4.0", "4.3.3"], required_python_versions = ["3.12.0", "3.11.0"] in Components section.

5. Selftest strengthened — Assert on scaffolded content: @connect @workbench tags, pytest_bdd imports, fixture definitions (not just file existence).

Bonus: Added just check to Verification section per CLAUDE.md; confirmed examples/ in ruff paths.

…8e494718eeb8a6

posit-vip-triage Bot requested review from bdeitte, ian-flores and statik as code owners May 29, 2026 18:30

This was referenced May 29, 2026

[Feature] add example of adding a custom test extension #298

Open

[aw] No-Op Runs #158

Open

ian-flores reviewed May 29, 2026

View reviewed changes

Copilot started work on behalf of ian-flores May 29, 2026 18:42 View session

ian-flores reviewed May 29, 2026

View reviewed changes

docs: confirm cross_product_validation naming, update all references

52ce242

Co-authored-by: ian-flores <18703558+ian-flores@users.noreply.github.com>

Copilot finished work on behalf of ian-flores May 29, 2026 18:44

Copilot AI requested a review from ian-flores May 29, 2026 18:44

Copilot started work on behalf of ian-flores May 29, 2026 18:44 View session

docs: address naming consistency and multi-template feedback

2a445e1

Co-authored-by: ian-flores <18703558+ian-flores@users.noreply.github.com>

Copilot finished work on behalf of ian-flores May 29, 2026 18:47

Copilot started work on behalf of ian-flores June 8, 2026 22:41 View session

Copilot finished work on behalf of ian-flores June 8, 2026 22:46

ian-flores mentioned this pull request Jun 8, 2026

docs: plan for #301 — add in-session execution primitives for Workbench #317

Merged

ian-flores added 2 commits June 9, 2026 13:36

Merge remote-tracking branch 'origin/main' into bot-plan-issue-298-df…

bbae9e6

…8e494718eeb8a6

docs: fix custom-test example plan (fixtures, selftest path, markers)

308b3fe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: plan for #298 — custom test extension scaffolding#299

docs: plan for #298 — custom test extension scaffolding#299
posit-vip-triage[bot] wants to merge 6 commits into
mainfrom
bot-plan-issue-298-df8e494718eeb8a6

posit-vip-triage Bot commented May 29, 2026 •

edited by Copilot AI

Loading

Uh oh!

github-actions Bot commented May 29, 2026

Uh oh!

ian-flores May 29, 2026

Uh oh!

Copilot AI May 29, 2026

Uh oh!

ian-flores May 29, 2026

Uh oh!

Copilot AI May 29, 2026

Uh oh!

ian-flores May 29, 2026

Uh oh!

Copilot AI May 29, 2026

Uh oh!

ian-flores commented Jun 8, 2026

Uh oh!

Copilot AI commented Jun 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		- UNCONFIRMED: Should the example test directory be named `examples/gxp_validation/` or something more generic like `examples/cross_product_validation/`?
		- Leaning toward: `examples/cross_product_validation/` because the pattern applies beyond GxP (any customer who needs to lock runtime/package versions).

		- UNCONFIRMED: Should the scaffold command default to copying from `examples/cross_product_validation/` or should it be extensible (multiple scaffold templates)?
		- Leaning toward: single template for now; extensibility can be added later if demand emerges.


		## Out of scope

		- Modifying the existing `examples/custom_tests/` directory (it serves a different purpose: minimal HTTP check). The new example is complementary, not a replacement.

Conversation

posit-vip-triage Bot commented May 29, 2026 • edited by Copilot AI Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 29, 2026

Uh oh!

ian-flores May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 29, 2026

Choose a reason for hiding this comment

Uh oh!

ian-flores May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 29, 2026

Choose a reason for hiding this comment

Uh oh!

ian-flores May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI May 29, 2026

Choose a reason for hiding this comment

Uh oh!

ian-flores commented Jun 8, 2026

Uh oh!

Copilot AI commented Jun 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

posit-vip-triage Bot commented May 29, 2026 •

edited by Copilot AI

Loading