Extension System Testing

Overview
Unified Testing Architecture
CI Environment Setup
Test Coverage
Workflow Triggers
Resource Requirements
Interpreting Results
Adding New Extensions
Best Practices

Overview

The unified integration testing workflow integration.yml provides comprehensive automated validation for the entire Sindri system, including all extensions in the docker/lib/extensions.d/ directory.

This testing ensures that users can confidently activate and use any extension through the extension-manager command without encountering issues. Each extension implements the Extension API v1.0 or v2.0 (see EXTENSIONS.md) with 6-7 required functions.

Testing Philosophy

The unified test suite is designed to:

Validate All API Functions - Test all Extension API functions for compliance
Test Complete Integration - Each extension runs API compliance AND integration tests
Ensure Reliability - Idempotency, error handling, and conflict detection
Maintain Quality - Syntax validation, best practices, and documentation
Controlled Concurrency - Max 3 jobs concurrent to manage resources

Test Statistics

Main Workflow: 1 (integration.yml - single entry point)
Test Phases: 6 sequential phases (quick-checks → manager-validation + manifest-validation → dependency-chain → infrastructure → individual-extensions → extension-combinations)
Extensions Tested: 21 out of 21 (100%)
API Functions Coverage: 100% (6/6 v1.0, 7/7 v2.0)
Test Type: Complete (API compliance + integration + idempotency + upgrade)
Base System Components: 4 (mise, workspace, SSH, Claude Code CLI) - pre-installed and always available
Max Concurrent Jobs: 3 (controlled resource usage)
Validation Workflows: manager-validation.yml, manifest-validation.yml (comprehensive checks)

Unified Testing Architecture

Sequential Execution Flow

All testing flows through a single entry point (integration.yml) with 6 sequential phases:

Phase 1: quick-checks (inline, fail-fast)
    ↓
Phase 2: Extension System Validation (static, no VM, fail-fast)
    ├─→ manager-validation (extension manager validation)
    └─→ manifest-validation (comprehensive extension validation)
    ↓
Phase 2.5: dependency-chain (3 VMs, max 3 concurrent, fail-fast)
    ↓
Phase 3: infrastructure (1 VM, fail-fast)
    ↓
Phase 4: individual-extensions (N VMs, 3 max concurrent, fail-after-all-complete)
    ↓
Phase 5: extension-combinations (N VMs, 3 max concurrent, fail-after-all-complete)

Phase Descriptions

Phase 1: Quick Checks (2-5 minutes)

Shellcheck validation for all shell scripts
YAML syntax validation
Extension manager syntax check
Fail behavior: Stop immediately on error

Phase 2: Extension System Validation (3-4 minutes)

Manager Validation (1-2 minutes):

Extension manager script syntax validation
Extension name extraction testing
Manifest file operations testing
Directory-based naming convention verification

Manifest & Extension Validation (2 minutes):

Static extension syntax validation (basic)
Static dependency chain verification (files exist)
Manifest template validation
Extension metadata validation
Comprehensive shellcheck validation (all .extension and .sh files)
Common.sh sourcing verification
Shebang presence verification
Extension API v1.0/v2.0 function validation (all required functions)

Fail behavior: Stop immediately on error

Phase 2.5: Dependency Chain Tests (3-5 minutes)

Tests automatic dependency resolution (Extension API v2.2):

Test Cases:

nodejs-devtools → nodejs - TypeScript tools auto-install Node.js
openskills → nodejs + git - OpenSkills auto-installs Node.js and git
playwright → nodejs - Playwright auto-installs Node.js

Test Flow:

Create manifest with ONLY top-level extension (e.g., playwright)
Run extension-manager install-all
Verify dependencies automatically added to manifest
Verify correct install order (dependencies first)
Verify all extensions validated successfully

What's Tested:

Dependency detection from EXT_DEPENDENCIES metadata
Topological sort algorithm (correct install order)
Manifest auto-update with resolved dependencies
Installation in dependency-first order
Prevention of "prerequisite not met" errors

Resources: Max 3 concurrent VM deployments (standard tier: 4GB/2CPU)

Fail behavior: Stop immediately on error

Phase 3: Infrastructure Tests (5-7 minutes)

VM deployment and configuration (minimal tier: 1GB/1CPU)
SSH connectivity, volumes, persistence
Secrets management (SOPS + age)
Basic developer workflow
Fail behavior: Stop immediately on error

Phase 4: Individual Extensions (15-20 minutes)

Matrix of all 20+ extensions
Each extension runs complete test suite (see below)
Max 3 concurrent extension tests
Fail behavior: Let all running jobs complete, then fail

Phase 5: Extension Combinations (10-12 minutes)

Multi-extension scenario testing
Common stacks: mise-stack, full-node, fullstack, systems, enterprise, ai-dev
Max 3 concurrent combinations
Fail behavior: Let all running jobs complete, then fail

CI Environment Setup

Extension Auto-Installation Architecture

The CI testing environment uses a pre-installed base system with optional extension installation:

Pre-Installed Base System (Build Time)

Built into Docker image (Dockerfile):

The following components are baked into the Docker image and always available:

workspace-structure - /workspace directory layout
mise - Unified tool version manager
ssh-environment - Non-interactive SSH session support
claude-auth-with-api-key - Claude Code API key authentication (optional)

Benefits: ~10-12 second startup vs ~90-120 seconds with extension-based installation.

Optional Extensions (Runtime)

File: docker/lib/extensions.d/active-extensions.ci.conf

This manifest template defines additional extensions to install in CI mode.

Runtime Installation (Container Startup)

File: docker/scripts/entrypoint.sh

When the container starts, entrypoint.sh copies the extension library to the persistent volume:

Copy Library to Persistent Volume
```
if [ ! -d "/workspace/.system/lib" ]; then
    cp -r /docker/lib /workspace/scripts/
```
- Only runs on first boot (volume empty)
- Copies extension library and manager from Docker image

Select Appropriate Manifest

if [ "$CI_MODE" = "true" ]; then
    cp /docker/lib/extensions.d/active-extensions.ci.conf \
       /workspace/scripts/.system/manifest/active-extensions.conf

CI mode: Uses CI template
Production mode: Uses existing manifest or default

Optional Extension Installation (test-specific)

Tests that require additional extensions (nodejs, python, rust, etc.) use extension-manager install-all to install them from the manifest.

Why Pre-Install Base System?

Performance:

✅ 10-12 seconds startup vs 90-120 seconds with extension-based installation
✅ 75% faster CI/CD workflows
✅ Immediate availability of core tools

Reliability:

✅ No DNS/network failures during startup
✅ Consistent versions across all instances
✅ Reduced complexity in entrypoint scripts

Deployment Flow Diagram

Docker Build
    │
    ├─→ Installs base system (workspace, mise, ssh, Claude Code CLI)
    ├─→ Includes active-extensions.ci.conf (template)
    │
    ▼
Container Startup (entrypoint.sh)
    │
    ├─→ Base system: ✓ Already available (mise, workspace, SSH, Claude Code CLI)
    │
    ├─→ Check: First boot? (/workspace/.system/lib missing?)
    │   │
    │   ├─→ YES: Copy lib/ to /workspace/scripts/
    │   │         Copy active-extensions.ci.conf → active-extensions.conf
    │   │
    │   └─→ NO: Skip (volume already has lib/)
    │
    ▼
Container Ready
    │
    ├─→ Base system: ✓ Available (mise, workspace, SSH, Claude Code CLI)
    └─→ Optional: Tests install additional extensions as needed

Testing Implications

Workflow Verification Steps (integration.yml, extension-tests.yml):

After deployment, workflows verify the base system:

# Verify base system components
mise --version
ls /workspace/projects
which ssh
claude --version

# Verify extension library copied to volume
ls /workspace/.system/bin/extension-manager

Key Differences:

Base System: Always available immediately (pre-installed in Docker image)
Extensions: Installed on-demand via extension-manager install-all
CI: Fresh volume every test
Production: Persistent volume

Debugging Base System Issues

Verify base system is available:

which mise
# Expected: /usr/local/bin/mise

ls -la /workspace
# Expected: projects/ scripts/ config/ developer/ docs/

which ssh
# Expected: /usr/local/bin/ssh (wrapper script)

which claude
# Expected: /usr/local/bin/claude

If components missing:

Docker image may not be built correctly
Check Dockerfile for base system installation steps
Rebuild Docker image: fly deploy or CI build workflow

Test Coverage

Complete Extension Test Suite

Each extension in the individual-extensions phase runs a complete test suite via the test-extension-complete composite action:

1. Installation (with Dependencies)

Adds extension and dependencies to manifest
Runs extension-manager install-all
Verifies installation completed without errors

2. Validation

Executes extension-manager validate <extension>
Ensures post-install checks pass

3. API Compliance

Tests validate() function returns success
Tests status() function shows required fields (Extension, Status)
Verifies extension is in manifest

4. Key Functionality

Tests primary tool functionality (tool-specific)
Verifies commands are available in PATH
Runs basic smoke tests

5. Idempotency

Runs extension-manager install-all a second time
Verifies no errors on re-installation
For claude-marketplace: Verifies settings.json unchanged

6. Upgrade (API v2.0+)

Tests upgrade() function if present
Runs upgrade dry-run
Verifies upgrade capability

Test Script Organization

All test scripts are organized by concern in .github/scripts/:

Shared Libraries (`lib/`)

test-helpers.sh - 20+ utility functions
assertions.sh - 10+ test assertions
ssh-helpers.sh - SSH/SFTP with retry logic
retry-utils.sh - Backoff and retry utilities

Infrastructure Tests (`infrastructure/`)

test-vm-deployment.sh - VM, SSH, volumes, extension system
test-secrets-management.sh - SOPS + age encryption
verify-volume.sh - Volume mount verification

Extension Tests (`extensions/`)

test-extension-complete.sh - Complete test suite (consolidated)
verify-commands.sh - Command availability
test-key-functionality.sh - Tool-specific tests (30+ tools)

Validation (`validation/`)

verify-manifest.sh - Manifest validation
check-syntax.sh - Extension syntax
verify-dependencies.sh - Dependency chains
setup-manifest.sh - Manifest utilities

Composite Actions

New composite actions streamline extension testing:

test-extension-complete - Runs complete test suite for one extension
install-extension - Installs extension with dependencies
upload-test-libs - Uploads shared test libraries to VM

For complete Extension API specification, see EXTENSIONS.md - Extension API Specification.

Tested Extensions (21 Total)

Note: The base system (workspace-structure, mise, ssh-environment, Claude Code CLI) is pre-installed in the Docker image and not tested as extensions.

Extension	Key Tools	Dependencies	Test Focus
Languages (mise-powered)
nodejs	node, npm	mise (pre-installed)	Runtime, package manager
python	python3, pip3	mise (pre-installed)	Execution, packages
rust	rustc, cargo	mise (pre-installed)	Compilation, cargo
golang	go	mise (pre-installed)	Compilation, modules
nodejs-devtools	tsc, eslint	mise (pre-installed), nodejs	TypeScript, linting
ruby	ruby, gem, bundle	mise (pre-installed)	Ruby execution, Rails
Languages (Traditional)
php	php, composer	-	PHP, Symfony
jvm	java, sdk	-	SDKMAN, Java
dotnet	dotnet	-	.NET SDK, ASP.NET
Claude AI
claude-marketplace	marketplace config	git	Plugin installation
openskills	openskills, skills	nodejs, git	Skills management
Infrastructure
docker	docker, compose	-	Container runtime
infra-tools	terraform, ansible	-	IaC tools
cloud-tools	aws	-	Cloud CLIs
ai-tools	codex, gemini, droid	nodejs	AI assistants (incl. Factory AI Droid)
Utilities
playwright	playwright	nodejs	Browser automation
monitoring	claude-monitor	python, nodejs	Usage tracking
tmux-workspace	tmux	-	Session management
agent-manager	agent-manager	-	Agent management
context-loader	context-load	-	Context utilities
github-cli	gh	-	GitHub CLI

Test Steps

For each extension using extension-manager:

Dependency Installation: Auto-install dependencies based on depends_on field
Manifest Addition: Add extension to active-extensions.conf
Installation: Run extension-manager install-all
Command Availability: Verify all expected commands in PATH
mise Verification: For mise-powered extensions, verify managed by mise
Key Functionality: Test core capability
Status Check: Verify extension-manager status <name> output
Idempotency: Re-run installation to verify safe re-execution
Resource Cleanup: Destroy test VM and volumes

When It Runs:

On push/PR affecting extension files
On workflow dispatch (all or specific extension)

4. Extension API Tests (CRITICAL)

Tests all Extension API functions for compliance.

For complete Extension API specification, see EXTENSIONS.md - Extension API Specification.

Functions Tested

validate(): Verifies installed extension passes validation checks
status(): Confirms output includes Extension name and Status fields
uninstall(): Tests that remove() function is called and cleans up properly
deactivate(): Verifies extension is removed from manifest
upgrade() (v2.0): Tests upgrade functionality for API v2.0 extensions

Test Matrix

Representative sample of 9 extensions tested for full API compliance:

nodejs (mise-powered language)
python (mise-powered language)
rust (mise-powered language)
golang (mise-powered language)
tmux-workspace (native apt-based utility)
monitoring (apt packages with multi-dependency)
docker (mixed: apt + binary downloads, complex multi-step)
openskills (npm global install pattern)
agent-manager (git/GitHub release binary pattern)

When It Runs: On every push/PR affecting extension files

5. Base System Verification Tests

Tests that the pre-installed base system is available and functional.

For details on the base system, see EXTENSIONS.md - Pre-Installed Base System.

Verification Tests

mise Availability: Verifies mise --version works
Workspace Structure: Verifies /workspace directory tree exists
SSH Environment: Verifies SSH wrapper scripts are in place
Claude CLI: Verifies claude --version works

When It Runs: On every push/PR affecting extension files or Docker image

6. Manifest Operations Tests

Tests manifest file operations and integrity:

Operations Tested

Reorder Functionality:
- Tests extension-manager reorder <name> <position>
- Verifies extension moves to exact position
- Uses test fixture with 6 extensions
Comment Preservation:
- Tests deactivate preserves user comments in manifest
- Verifies header comments, section comments, inline comments preserved
- Uses test fixture with multiple comment types

Test Fixtures Used:

.github/workflows/test-fixtures/manifest-reorder-test.conf
.github/workflows/test-fixtures/manifest-with-comments.conf

When It Runs: On every push/PR affecting extension files

8. Dependency Chain Tests

Tests dependency resolution and error handling:

Dependency Tests

Transitive Dependencies:
- Tests multi-level dependency chain (nodejs-devtools → nodejs)
- Adds only top-level extension to manifest
- Verifies all dependencies auto-install
- Uses test fixture with single extension
Missing Dependency Errors:
- Simulates mise unavailability to break prerequisites
- Attempts to install nodejs (which requires mise)
- Verifies installation fails with clear error message
- Confirms error mentions "prerequisite" or "mise required"

Test Fixtures Used: .github/workflows/test-fixtures/manifest-only-top-level.conf

When It Runs: On every push/PR affecting extension files

9. Extension Combinations

Tests common extension combinations for conflicts using manifest-based activation:

Test Combinations

Each combination activates multiple extensions in active-extensions.conf:

Note: Base system (workspace, mise, ssh, Claude Code CLI) is pre-installed and not included in manifest.

mise-stack: nodejs, python, rust, golang (mise-Powered Languages)
full-node: nodejs, nodejs-devtools (Complete Node.js Development Stack)
fullstack: nodejs, python, docker, cloud-tools (Python + Docker + Cloud)
systems: rust, golang, docker (Rust + Go + Docker)
enterprise: nodejs, jvm, docker, infra-tools (JVM + Docker + Infrastructure)
ai-dev: nodejs, python, ai-tools, monitoring (Python + AI Tools + Monitoring)

Validation

All extensions activate successfully via extension-manager install-all
Base system (mise, workspace, SSH, Claude Code CLI) is already available
No installation conflicts between extensions
Cross-extension functionality works
Tools from different extensions coexist
Dependencies are properly resolved

When It Runs:

Manual workflow dispatch
Commit messages containing [test-combinations]

10. Results Reporting

Generates comprehensive test report summary:

Job status for all test categories
Success/failure indicators
Links to detailed logs
GitHub Actions summary

Test Fixtures

To avoid complex heredoc escaping in GitHub Actions workflows, test fixtures are used for manifest testing:

Available Fixtures

Located in .github/workflows/test-fixtures/:

Fixture File	Purpose	Used By
`manifest-reorder-test.conf`	6 extensions for reorder testing	manifest-operations-tests
`manifest-with-comments.conf`	Manifest with various comment types	manifest-operations-tests
`manifest-only-top-level.conf`	Single top-level extension for dependency testing	dependency-chain-tests

Benefits of Test Fixtures

No Escaping Issues: Avoid complex quote/heredoc escaping in nested SSH commands
Version Controlled: Fixtures tracked in git alongside tests
Reusable: Same fixture can be used across multiple test scenarios
Maintainable: Easy to update test data without touching workflow YAML
Readable: Clear separation between test logic and test data

Using Fixtures in Tests

- name: Copy test fixture to VM
  run: |
    flyctl ssh sftp shell --app $app_name <<'SFTP_EOF'
      put .github/workflows/test-fixtures/manifest-reorder-test.conf /tmp/manifest-reorder.conf
      bye
    SFTP_EOF

- name: Test with fixture
  run: |
    flyctl ssh console --app $app_name --command "/bin/bash -lc '
      cp /tmp/manifest-reorder.conf .system/manifest/active-extensions.conf
      # Run tests...
    '"

Workflow Triggers

Automatic Triggers

The workflow runs automatically in two scenarios:

Pull Request Trigger

pull_request:
  branches: [main, develop]
  paths:
    # Core extension system files only (not individual extensions)
    - "docker/lib/extension-manager.sh"
    - "docker/lib/common.sh"
    - "docker/lib/extensions-common.sh"
    # Workflows and test infrastructure
    - ".github/workflows/extension-tests.yml"
    - ".github/workflows/per-extension.yml"
    - ".github/workflows/api-compliance.yml"
    # Test scripts
    - ".github/scripts/extension-tests/**"

When It Runs: Only when core extension infrastructure changes (not individual extension modifications)

Scheduled Trigger

schedule:
  - cron: '0 2 * * 0'  # Weekly on Sunday at 2:00 AM UTC

Rationale: Regular validation ensures:

All extensions remain functional
Early detection of upstream dependency issues
Validation of extension interactions

Manual Triggers

For on-demand testing and debugging:

# Test all extensions (full test suite)
gh workflow run extension-tests.yml

# Test specific extension
gh workflow run extension-tests.yml \
  -f extension_name=rust \
  -f skip_cleanup=false

# Test all extensions with cleanup disabled (for debugging)
gh workflow run extension-tests.yml \
  -f skip_cleanup=true

# Test with idempotency checks disabled (faster)
gh workflow run extension-tests.yml \
  -f skip_idempotency=true

# Test in specific region
gh workflow run extension-tests.yml \
  -f region=lhr

Available Parameters:

extension_name (optional): Test specific extension (e.g., rust, golang, ai-tools)
skip_cleanup (boolean, default: false): Skip cleanup for debugging
skip_idempotency (boolean, default: false): Skip idempotency tests for faster runs
test_app_prefix (string, default: "ext-test"): Custom app name prefix
region (choice, default: "sjc"): Fly.io region for testing

Trigger Strategy

What Changed:

❌ Removed: Automatic push trigger (was running on every commit to main/develop)
✅ Kept: pull_request trigger for core infrastructure changes
✅ Kept: Weekly schedule trigger
✅ Kept: Manual workflow_dispatch trigger

Rationale:

Performance: Full test suite takes 45-60 minutes (~20 VMs)
Cost: Avoid redundant runs on main branch (PR already validated)
Feedback: Pull requests still get validation before merge
Coverage: Weekly runs catch regressions in all extensions

Test Coverage Metrics

Extension API Coverage

API Function	Tested	Test Job	Coverage
`prerequisites()`	✅	per-extension-tests, dependency-chain-tests	100%
`install()`	✅	per-extension-tests, extension-api-tests	100%
`configure()`	✅	per-extension-tests	100%
`validate()`	✅	extension-api-tests	100%
`status()`	✅	per-extension-tests, extension-api-tests	100%
`remove()`	✅	extension-api-tests	100%
`upgrade()` (v2.0)	✅	extension-api-tests	100%

Overall API Coverage: 100% (7/7 functions including v2.0 and v2.1)

Feature Coverage

Feature	Tested	Test Job	Coverage
Base System Verification	✅	base-system-tests	100%
Cleanup Extensions Ordering	✅	cleanup-extensions-tests	100%
Dependency Resolution	✅	dependency-chain-tests	100%
Manifest Comment Preservation	✅	manifest-operations-tests	100%
Extension Reordering	✅	manifest-operations-tests	100%
Error Handling	✅	dependency-chain-tests	75%
Idempotency	✅	per-extension-tests	100%

Overall Feature Coverage: ~96%

Extension Coverage

Total Extensions: 21 (base system components not counted as extensions)
Extensions Tested: 21
Coverage: 100%
Untested: None (template intentionally excluded, used as development reference)

Resource Requirements

Resource Tier System

Sindri uses a 4-tier resource classification system to optimize VM resources based on extension requirements:

Tier	Memory	CPUs	CPU Kind	Disk	Use Case
Minimal	1GB	1	shared	10GB	Lightweight utilities, simple installations
Standard	4GB	2	shared	20GB	Standard packages, mise-managed runtimes
Heavy	8GB	4	performance	20GB	Compilation, containers, multiple runtimes
XHeavy	16GB	4	performance	30GB	Desktop environments, multiple extensions

Extension Resource Tier Classifications

Minimal Tier (1GB/1CPU):

tmux-workspace
context-loader
monitoring
agent-manager
github-cli

Standard Tier (4GB/2CPU):

nodejs
python
golang
nodejs-devtools
claude-marketplace
openskills
php
playwright

Heavy Tier (8GB/4CPU):

rust (compilation-heavy)
ruby (Rails + gems)
jvm (multiple JDK versions)
dotnet (.NET SDK)
docker (container runtime)
infra-tools (Terraform, Ansible, kubectl)
cloud-tools (AWS, Azure, GCP CLIs)
ai-tools (multiple AI CLIs including Factory AI Droid)

XHeavy Tier (16GB/4CPU):

xfce-ubuntu (desktop environment)
guacamole (remote desktop gateway)
extension-combinations (multiple extensions simultaneously)

Test Job VM Specifications

Test Type	Tier	Memory	CPUs	Disk	Timeout
Per-Extension	Variable	Variable	Variable	Variable	60 min
Extension API Tests	Variable	Variable	Variable	Variable	45 min
Base System Tests	Minimal	1GB	1	10GB	40 min
Cleanup Extensions	Minimal	1GB	1	10GB	35 min
Manifest Operations	Standard	4GB	2	20GB	35 min
Dependency Chain	Standard	4GB	2	20GB	50 min
Combinations	XHeavy	16GB	4	30GB	90 min

Note: Per-Extension and Extension API Tests use resource tiers dynamically based on the extension being tested.

Cost Considerations

Each VM deployment costs according to Fly.io pricing
Tests run in CI_MODE (SSH daemon disabled) for faster deployment
Automatic cleanup prevents lingering resources
Use skip_cleanup=true only for debugging

Test Environments

All tests use:

CI_MODE: Enabled to prevent SSH port conflicts
Fly.io Region: sjc (US West)
Deployment Strategy: immediate (skip health checks)
Volume Encryption: Disabled for faster setup

Interpreting Results

Success Criteria

A test passes when:

✅ Extension activates without errors
✅ vm-configure.sh completes successfully
✅ All expected commands are available
✅ Key functionality tests pass
✅ Idempotency check succeeds

Common Failures

Failure Type	Likely Cause	Resolution
Configuration timeout	Extension takes too long	Increase timeout in matrix
Command not found	Installation incomplete	Check installation steps in extension
Idempotency failure	No existence check	Add `command_exists` checks
Conflict detected	Duplicate installations	Review extension interactions
Prerequisites failed	Missing dependency	Add dependency to `depends_on` field
mise not found	Base system issue	Check Docker image build, mise should be pre-installed
Dependency chain broken	Required extension missing	Ensure dependencies installed before extension

Debugging Failed Tests

Check Workflow Logs: Detailed output for each step
Review VM Logs: flyctl logs -a <app-name>
Run with Skip Cleanup: Keep VM alive for inspection
Test Locally: Activate extension on local test VM
Enable Debug Mode: Set DEBUG=true in extension script

Adding New Extensions

When adding a new extension, ensure it will pass all 10 test jobs.

For complete extension development guide, see EXTENSIONS.md - Creating Extensions.

1. Create Extension File

# Get into extensions directory
cd docker/lib/extensions.d

# Create extension directory
mkdir -p r

# Copy template
cp -r template/* r/

# Rename and edit the extension file
mv r/template.extension r/r.extension
vim r/r.extension

Required API Functions:

prerequisites() - Check system requirements
install() - Install packages and tools
configure() - Post-install configuration
validate() - Run smoke tests
status() - Check installation state
remove() - Uninstall and cleanup
upgrade() - Upgrade tools (API v2.0)

See EXTENSIONS.md - Extension API Specification for details.

2. Add to Test Matrix

Update .github/workflows/extension-tests.yml:

matrix:
  extension:
    # ... existing extensions ...
    - { name: "r", commands: "R,Rscript", key_tool: "R", timeout: "20m", uses_mise: "true" }

Note: mise is pre-installed, so no need to specify it as a dependency.

Matrix Fields:

name: Extension name (matches .extension filename)
commands: Comma-separated list of commands to verify
key_tool: Primary command for functionality testing
timeout: Max installation time
depends_on: Comma-separated dependencies (optional)
uses_mise: Set to 'true' if mise-powered (optional)
run_last: Set to 'true' for cleanup extensions (optional)

3. Add Functionality Test

In the workflow, add test case to Test key functionality step:

case "$key_tool" in
  # ... existing cases ...
  R)
    echo "Testing R..."
    R --version
    Rscript --version
    Rscript -e 'print("Hello from R")'
    ;;
esac

4. Test Locally First

# On test VM
cd /workspace/.system/lib

# Add to manifest
echo "r" >> .system/manifest/active-extensions.conf

# Install
bash extension-manager.sh install-all

# Validate
bash extension-manager.sh validate r

# Test all API functions
bash extension-manager.sh status r
bash extension-manager.sh uninstall r
bash extension-manager.sh deactivate r

5. Verify Passes All Checks

Job 1: Extension Manager Validation

Extension shows in extension-manager list
Name extraction works correctly

Job 2: Extension Syntax Validation

Shellcheck validation passes
All required API functions defined
Proper shebang and sourcing

Job 3: Per-Extension Tests

Installation completes within timeout
All commands available after installation
Key functionality test passes
Idempotent (safe to run multiple times)

Job 4: Extension API Tests (if in sample)

validate() returns 0
status() outputs correct format
uninstall() calls remove() correctly
deactivate() removes from manifest
upgrade() works (if API v2.0)

Job 5-8: Base System/Cleanup/Manifest/Dependencies

Base system (mise, workspace, SSH, Claude Code CLI) remains functional
Doesn't interfere with cleanup ordering
Works with manifest comment preservation
Dependencies correctly declared and installed

Best Practices

For Extension Developers

Always Check Existence: Use command_exists before installing
Handle Errors Gracefully: Don't exit on minor failures
Use Print Functions: print_status, print_success, print_error
Test Idempotency: Extension should be safe to run multiple times
Document Dependencies: Note any required extensions
Set Reasonable Timeouts: Consider installation time
Implement upgrade(): Support API v2.0 for upgrade functionality
Declare Required Domains (API v2.1+): Use EXT_REQUIRED_DOMAINS for domains needed during installation

DNS Check Best Practices (API v2.1+)

Declare domains needed for installation:

# Extension metadata
EXT_REQUIRED_DOMAINS="example.com github.com registry.npmjs.org"

# In prerequisites()
prerequisites() {
  print_status "Checking prerequisites for ${EXT_NAME}..."
  check_disk_space 1000
  check_required_domains || return 1
  print_success "All prerequisites met"
  return 0
}

Benefits: Automatic DNS checks, pre-flight aggregation, clear error messages, faster failure detection

See EXTENSIONS.md - Development Guidelines for complete guidelines.

For Extension Users

Review Test Results: Check workflow before activating new extensions
Test Individually: Activate one extension at a time initially
Check Combinations: Review combination tests for your stack
Monitor Resources: Extensions increase VM resource usage
Validate Installation: Run validation scripts after activation

Continuous Improvement

The extension testing system continuously evolves:

Metrics Tracked

Test execution time per extension
Success/failure rates
Resource usage patterns
Common failure modes
API function compliance
Dependency resolution success rates

Recent Enhancements (Completed)

Planned Enhancements

Performance benchmarking for extensions
Cross-platform testing (different VM sizes)
Circular dependency detection testing
Extension marketplace scoring
Installation time optimization tracking
Automated conflict detection across all combinations
Network reliability testing for DNS checks

Test Job Workflow

Execution Flow

┌─────────────────────────────────────────────────────────────┐
│  1. Extension Manager Validation (Quick Checks)             │
│     ↓ Validates extension-manager.sh script                 │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│  2. Extension Syntax Validation (Static Analysis)           │
│     ↓ Shellcheck all .extension files                       │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│  Jobs 3-9 Run in Parallel (Matrix Tests)                    │
├─────────────────────────────────────────────────────────────┤
│  3. Per-Extension Tests (20 extensions × install/validate)  │
│  4. Extension API Tests (9 extensions × all API functions)  │
│  5. Base System Tests (verification tests)                  │
│  6. Cleanup Extensions (ordering tests)                     │
│  7. Manifest Operations (reorder, comments)                 │
│  8. Dependency Chain (transitive deps, errors)              │
│  9. Extension Combinations (6 common stacks)                │
└─────────────────────────────────────────────────────────────┘
                           ↓
┌─────────────────────────────────────────────────────────────┐
│  10. Results Reporting (Aggregate Results)                  │
│      ↓ Summary of all test outcomes                         │
└─────────────────────────────────────────────────────────────┘

Critical Path

Must Pass Before Merge:

Job 1: Extension Manager Validation
Job 2: Extension Syntax Validation
Job 5: Base System Tests
Job 6: Cleanup Extensions Tests

Recommended Before Merge:

Job 3: Per-Extension Tests (for modified extensions)
Job 4: Extension API Tests (validates API compliance)
Job 7: Manifest Operations Tests (manifest integrity)
Job 8: Dependency Chain Tests (dependency resolution)

Optional (Manual Trigger):

Job 9: Extension Combinations (comprehensive stack testing)

Support

For issues with extension testing:

Review Logs: Check GitHub Actions workflow logs
Test Locally: Reproduce on your own test VM
Inspect Fixtures: Review test fixtures in .github/workflows/test-fixtures/
Open Issue: Report problems with test workflow
Contribute: Submit PRs to improve testing

FilesExpand file tree

EXTENSION_TESTING.md

Latest commit

History

EXTENSION_TESTING.md

File metadata and controls

Extension System Testing

Table of Contents

Overview

Testing Philosophy

Test Statistics

Unified Testing Architecture

Sequential Execution Flow

Phase Descriptions

Phase 1: Quick Checks (2-5 minutes)

Phase 2: Extension System Validation (3-4 minutes)

Phase 2.5: Dependency Chain Tests (3-5 minutes)

Phase 3: Infrastructure Tests (5-7 minutes)

Phase 4: Individual Extensions (15-20 minutes)

Phase 5: Extension Combinations (10-12 minutes)

CI Environment Setup

Extension Auto-Installation Architecture

Pre-Installed Base System (Build Time)

Optional Extensions (Runtime)

Runtime Installation (Container Startup)

Why Pre-Install Base System?

Deployment Flow Diagram

Testing Implications

Debugging Base System Issues

Test Coverage

Complete Extension Test Suite

1. Installation (with Dependencies)

2. Validation

3. API Compliance

4. Key Functionality

5. Idempotency

6. Upgrade (API v2.0+)

Test Script Organization

Shared Libraries (lib/)

Infrastructure Tests (infrastructure/)

Extension Tests (extensions/)

Validation (validation/)

Composite Actions

Tested Extensions (21 Total)

Test Steps

4. Extension API Tests (CRITICAL)

Functions Tested

Test Matrix

5. Base System Verification Tests

Verification Tests

6. Manifest Operations Tests

Operations Tested

8. Dependency Chain Tests

Dependency Tests

9. Extension Combinations

Test Combinations

Validation

10. Results Reporting

Test Fixtures

Available Fixtures

Benefits of Test Fixtures

Using Fixtures in Tests

Workflow Triggers

Automatic Triggers

Pull Request Trigger

Scheduled Trigger

Manual Triggers

Trigger Strategy

Test Coverage Metrics

Extension API Coverage

Overall API Coverage: 100% (7/7 functions including v2.0 and v2.1)

Feature Coverage

Overall Feature Coverage: ~96%

Extension Coverage

Resource Requirements

Resource Tier System

Extension Resource Tier Classifications

Test Job VM Specifications

Cost Considerations

Test Environments

Shared Libraries (`lib/`)

Infrastructure Tests (`infrastructure/`)

Extension Tests (`extensions/`)

Validation (`validation/`)