[PyTorch] [CI] Capture subprocess stderr in distributed tests for better CI error re…

[sudhakarsingh27]

…porting

Distributed tests launch subprocesses via torch.distributed.launch/torchrun. When these fail, pytest only captures the CalledProcessError from the parent process, not the actual worker traceback. This makes CI JUnit XML reports show "exit code 1" with no useful error detail.

Add run_distributed() utility to tests/pytorch/utils.py that captures stderr while letting stdout stream to the terminal. On failure, the worker's stderr (containing the actual Python traceback) is included in the AssertionError, which pytest writes into the JUnit XML report.

Behavior:

• Interactive use: stdout streams in real time (unchanged), stderr shown on failure
• CI/JUnit XML: failure reports now include the actual worker traceback

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

• Documentation change (change only to the documentation, either a fix or a new content)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Infra/Build change
• Code refactoring

Changes

Please list the changes introduced in this PR:

• Change A
• Change B

Checklist:

• I have read and followed the contributing guidelines
• The functionality is complete
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[greptile-apps]

Greptile Summary

This PR improves CI error reporting for distributed PyTorch tests by introducing a run_distributed() helper in tests/pytorch/utils.py that captures subprocess stderr and includes it in AssertionError messages (and therefore JUnit XML reports) on failure, while leaving stdout to stream live to the terminal. Six test files are migrated from raw subprocess.run(..., check=True) calls to this new utility, and qa/L0_cppunittest/test.sh gains JUnit XML output for C++ unit tests.

Key changes:

• tests/pytorch/utils.py: New run_distributed() wraps subprocess.run with stderr=subprocess.PIPE, text=True; on non-zero exit it raises AssertionError with up to 4 000 chars of the stderr tail, making worker tracebacks visible in JUnit XML.
• valid_returncodes parameter: Callers that run inner pytest processes (e.g. FSDP2 tests) can pass valid_returncodes=(0, 5) to treat pytest's "all-skipped" exit code as success.
• qa/L0_cppunittest/test.sh: Adds --output-junit to ctest so C++ unit test results also appear in JUnit XML reports.
• One unused import subprocess remains in test_torch_fsdp2.py after the migration and should be cleaned up.

Confidence Score: 5/5

Safe to merge — all remaining findings are minor style issues that do not affect correctness or CI reliability.

The core utility is well-designed: stderr is captured for reporting, stdout streams live, truncation logic is correct, and valid_returncodes handles the inner-pytest exit-5 case cleanly. The only issue found is an unused import subprocess left in test_torch_fsdp2.py, which is P2 style only.

tests/pytorch/distributed/test_torch_fsdp2.py — unused import subprocess should be removed.

Important Files Changed

Filename	Overview
tests/pytorch/utils.py	Adds run_distributed() helper that captures stderr via subprocess.PIPE while leaving stdout to stream live; raises AssertionError with a 4000-char stderr tail on failure.
tests/pytorch/distributed/test_torch_fsdp2.py	Migrates two subprocess.run calls to run_distributed with valid_returncodes=(0,5); leaves an unused import subprocess that should be removed.
tests/pytorch/attention/test_attention_with_cp.py	Replaces two subprocess.run(..., check=True) calls with run_distributed(); no functional changes to test logic.
tests/pytorch/distributed/test_cast_master_weights_to_fp8.py	Adds sys.path manipulation and run_distributed import; replaces subprocess.run(command, check=True) with run_distributed(command).
tests/pytorch/distributed/test_fusible_ops_with_userbuffers.py	Adds run_distributed to imports; replaces subprocess.run(..., check=True, env=env) with run_distributed(command, env=env).
qa/L0_cppunittest/test.sh	Adds XML_LOG_DIR setup and passes --output-junit to ctest for JUnit XML reporting in CI.

Sequence Diagram

sequenceDiagram
    participant PT as pytest (parent)
    participant RD as run_distributed()
    participant SP as subprocess.run
    participant WK as torchrun / worker processes

    PT->>RD: call run_distributed(args, **kwargs)
    RD->>SP: subprocess.run(args, stderr=PIPE, text=True, **kwargs)
    SP->>WK: launch torchrun / distributed workers
    WK-->>SP: stdout → inherited (streams to terminal)
    WK-->>SP: stderr → PIPE (buffered in memory)
    SP-->>RD: CompletedProcess(returncode, stderr)

    alt returncode in valid_returncodes
        RD-->>PT: return CompletedProcess (success)
    else returncode not in valid_returncodes
        RD->>RD: build msg with last 4000 chars of stderr
        RD-->>PT: raise AssertionError(msg)
        PT->>PT: write AssertionError + stderr tail to JUnit XML
    end

Loading

_{Reviews (3): Last reviewed commit: "Merge branch 'main' into sudhakars/impro..." | Re-trigger Greptile}

[greptile-apps]

**kwargs can silently conflict with stderr and text

If a caller ever passes stderr= or text= through **kwargs, Python will raise TypeError: subprocess.run() got multiple values for keyword argument 'stderr'. Consider explicitly popping or blocking those keys, or documenting the restriction:

kwargs.pop("stderr", None)  # always captured internally
kwargs.pop("text", None)    # always text mode internally
result = subprocess.run(args, stderr=subprocess.PIPE, text=True, **kwargs)

None of the current call sites pass these, so this is not an immediate bug — just a fragile API surface.

[timmoon10]

/te-ci pytorch L1

[timmoon10]

LGTM