`sf agent test create` fails with invalid API Name error on Windows (works on macOS) but `--preview` + manual deploy works

[AndyK-huzz]

Summary

sf agent test create fails during "Deploying Metadata" step with "API Name" validation error, even when all names are valid. The same metadata deploys successfully using --preview followed by manual sf project deploy start, proving the generated metadata is valid. This appears to be a bug in how the command internally deploys metadata, possibly related to namespaced orgs.
This appears to be a Windows-specific issue - colleagues on macOS do not encounter this error with the same command and test spec.

Steps To Reproduce

Unable to provide public repo as this involves a namespaced scratch org. However, issue is 100% reproducible:

Create a namespaced scratch org (tested with a custom namespace)
Have an agent deployed to the org
Create a minimal test spec file specs/minimal_test-testSpec.yaml:

name: MinTest
description: Simple test
subjectType: AGENT
subjectName: My_Agent
testCases:

• utterance: Hello
  expectedTopic: My_Topic
  expectedActions:
  • QueryRecords
    expectedOutcome: A response
    customEvaluations: []
    conversationHistory: []
    metrics:
  • completeness
    Run: sf agent test create --spec specs/minimal_test-testSpec.yaml --api-name MinTest -o your-namespaced-org --force-overwrite
    Command fails at "Deploying Metadata" step
    Workaround proving metadata is valid:

Step 1: Generate preview (works)
sf agent test create --spec specs/minimal_test-testSpec.yaml --api-name MinTest -o your-org --preview

Step 2: Copy to proper location
cp MinTest-preview.xml force-app/main/default/aiEvaluationDefinitions/MinTest.aiEvaluationDefinition-meta.xml

Step 3: Manual deploy (works)
sf project deploy start --source-dir force-app/main/default/aiEvaluationDefinitions/MinTest.aiEvaluationDefinition-

Expected result

The sf agent test create command should successfully deploy the AiEvaluationDefinition metadata to the org, identical to how sf project deploy start succeeds with the same generated XML.

Actual result

Error (SfError): Name: The AI Evaluation Definition API Name can only contain underscores and alphanumeric characters. It must be unique, begin with a letter, not include spaces, not end with an underscore, and not contain two consecutive underscores.

✔ Creating Local Metadata
◯ Waiting for the org to respond - Skipped
✘ Deploying Metadata
Error occurs 100% of the time regardless of:

API name used (tried: MinTest, MinimalTest, SmartActions_Test)
Whether overwriting existing test or creating new
Running from short vs long path
Whether name in YAML matches --api-name flag

Additional information

sf doctor passes all diagnostics
Org has existing agent tests that were deployed successfully ~10 days ago (before CLI update)
Tested on namespaced scratch org
Terminal: Powershell on Windows

System Information

Shell: PowerShell

{
  "architecture": "win32-x64",
  "cliVersion": "@salesforce/cli/2.122.6",
  "nodeVersion": "node-v20.19.3",
  "osVersion": "Windows_NT 10.0.26100",
  "rootPath": "C:\\Users\\akennedy\\AppData\\Roaming\\nvm\\v20.19.3\\node_modules\\@salesforce\\cli",
  "shell": "PowerShell",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.2.40 (core)",
    "@oclif/plugin-commands 4.1.40 (core)",
    "@oclif/plugin-help 6.2.37 (core)",
    "@oclif/plugin-not-found 3.2.74 (core)",
    "@oclif/plugin-plugins 5.4.55 (core)",
    "@oclif/plugin-search 1.2.37 (core)",
    "@oclif/plugin-update 4.7.18 (core)",
    "@oclif/plugin-version 2.2.36 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.55 (core)",
    "@oclif/plugin-which 3.2.43 (core)",
    "@salesforce/cli 2.122.6 (core)",
    "agent 1.27.3 (core)",
    "apex 3.9.5 (core)",
    "api 1.3.8 (core)",
    "auth 4.1.4 (core)",
    "data 4.0.67 (core)",
    "deploy-retrieve 3.24.7 (core)",
    "info 3.4.100 (core)",
    "limits 3.3.74 (core)",
    "marketplace 1.3.8 (core)",
    "org 5.9.64 (core)",
    "packaging 2.24.11 (core)",
    "schema 3.3.90 (core)",
    "settings 2.4.54 (core)",
    "sobject 1.4.82 (core)",
    "telemetry 3.6.71 (core)",
    "templates 56.4.7 (core)",
    "trust 3.7.113 (core)",
    "user 3.6.48 (core)"
  ]
}

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.