Skip to content

Fix: propagate opentelemetry-instrument and ddtrace-run to flow run subprocesses#21041

Open
shnhdan wants to merge 10 commits intoPrefectHQ:mainfrom
shnhdan:fix/otel-flow-run-subprocess
Open

Fix: propagate opentelemetry-instrument and ddtrace-run to flow run subprocesses#21041
shnhdan wants to merge 10 commits intoPrefectHQ:mainfrom
shnhdan:fix/otel-flow-run-subprocess

Conversation

@shnhdan
Copy link

@shnhdan shnhdan commented Mar 9, 2026

Closes #20535

Problem

When users run opentelemetry-instrument prefect flow-run execute, the
auto-instrumentation setup does not propagate to the subprocess that
actually runs the flow. This is because the child process is spawned
fresh and doesn't inherit the instrumentation bootstrapped in the parent.

Solution

The process worker now detects active wrapper commands via environment
signals and prepends them to the subprocess command:

  • OTEL_* env vars → prepends opentelemetry-instrument
  • DD_* env vars → prepends ddtrace-run

Changes

  • Added _detect_active_wrappers() helper in process.py to detect
    active wrappers via env var signals
  • Updated prepare_for_flow_run() to prepend detected wrappers to the
    subprocess command
  • Added PREFECT_WORKER_PROCESS_COMMAND_WRAPPERS setting (default:
    ["opentelemetry-instrument", "ddtrace-run"]) so users can opt in/out
    or add custom wrappers by configuring this setting

Testing

  • 11 unit tests covering all wrapper detection scenarios
  • Verified manually via verify_otel.py

shnhdan added 3 commits March 9, 2026 16:48
When OTEL_* or DD_* env vars are detected, prepend the corresponding
wrapper command (opentelemetry-instrument or ddtrace-run) to the
subprocess command so auto-instrumentation propagates to child processes.

Adds PREFECT_WORKER_PROCESS_COMMAND_WRAPPERS setting to configure
which wrappers to detect, defaulting to opentelemetry-instrument
and ddtrace-run.

Closes PrefectHQ#20535
@github-actions github-actions bot added the bug Something isn't working label Mar 9, 2026
@codspeed-hq
Copy link

codspeed-hq bot commented Mar 9, 2026

Merging this PR will not alter performance

✅ 2 untouched benchmarks


Comparing shnhdan:fix/otel-flow-run-subprocess (87e2978) with main (98309ff)

Open in CodSpeed

Copy link
Member

@desertaxle desertaxle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for opening a PR @shnhdan!

As I mentioned on the linked issue, we'd prefer to use the setup_environment hook if possible to handle setting up OTel for the flow run execution process. Would you be able to try that approach instead?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prefect flow runs lose automatic opentelemetry-instrument configuration

2 participants