Skip to content

.NET: Bugfix Emit execute_tool spans when Auto Wiring via Agent UseOpenTelemetry#6667

Merged
peibekwe merged 1 commit into
microsoft:mainfrom
rogerbarreto:rogerbarreto/otel-below-ficc-tool-spans
Jun 22, 2026
Merged

.NET: Bugfix Emit execute_tool spans when Auto Wiring via Agent UseOpenTelemetry#6667
peibekwe merged 1 commit into
microsoft:mainfrom
rogerbarreto:rogerbarreto/otel-below-ficc-tool-spans

Conversation

@rogerbarreto

Copy link
Copy Markdown
Member

Motivation & Context

When an agent is instrumented with UseOpenTelemetry() and invokes in-proc tools, no execute_tool <name> span was emitted. The auto-wire placed OpenTelemetryChatClient above FunctionInvokingChatClient (FICC), producing OTel(FICC(leaf)). With OpenTelemetry above FICC, the chat span stays open during the tool loop, so FICC sees no invoke_agent source and skips the tool span. Tool calls only showed up inside gen_ai.output.messages, never as their own span, breaking parity with the GenAI semantic conventions and with Python.

This repositions OpenTelemetry below FICC, producing FICC(OTel(leaf)), so the tool span is emitted on the agent source.

Description & Review Guide

  • What are the major changes?

    • New internal DeferredOpenTelemetryChatClient: an inert DelegatingChatClient slot whose Activate(sourceName) swaps its target to inner.AsBuilder().UseOpenTelemetry(sourceName).Build(). Idempotent and lock guarded.
    • WithDefaultAgentMiddleware always registers this slot as the innermost decorator, so it lands directly below FICC.
    • OpenTelemetryAgent activates the slot once at construction (same guards as before: auto wire enabled, inner ChatClientAgent reachable, not UseProvidedChatClientAsIs, not already instrumented) and forwards run options straight through. The previous per-run ChatClientFactory outer wrap was removed.
    • Unit tests updated and added, including a proof that execute_tool is emitted on the agent source, parented under invoke_agent, plus a focused test for the inert/active slot.
  • What is the impact of these changes?

    • execute_tool <name> spans now appear for tool-calling agents that use UseOpenTelemetry().
    • The emitted telemetry now matches a hand-wired FICC(OpenTelemetry(leaf)) pipeline (verified across placements: auto wire, bring your own OpenTelemetry chat client, and hand-wired agent all produce invoke_agent over chat + execute_tool with no doubled spans).
    • Behavioral note: the inner chat span now fires per model call rather than once per run. Bringing your own OpenTelemetry chat client does not double the chat span; the slot stays inert when an OpenTelemetryChatClient is already present.
  • Why below FICC? FICC resolves the tool span source per call as invokeAgentActivity?.Source ?? _activitySource. With OpenTelemetry below FICC the chat span closes before the tool loop, so Activity.Current is the invoke_agent span and its source drives execute_tool.

Related Issue

Fixes #6666

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change.

Copilot AI review requested due to automatic review settings June 22, 2026 16:53
@moonbox3 moonbox3 added the .NET Usage: [Issues, PRs], Target: .Net label Jun 22, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated Code Review

Reviewers: 5 | Confidence: 90% | Result: All clear

Reviewed: Correctness, Security Reliability, Test Coverage, Failure Modes, Design Approach


Automated review by rogerbarreto's agents

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes missing execute_tool <name> spans when .NET agents use UseOpenTelemetry() with in-proc tools by changing the decorator ordering so OpenTelemetry runs below FunctionInvokingChatClient (FICC). It does so by introducing an inert “slot” chat-client decorator that can be activated once, ensuring tool spans are emitted on the agent ActivitySource and parented under invoke_agent.

Changes:

  • Added DeferredOpenTelemetryChatClient, an inert DelegatingChatClient that can be activated to insert OpenTelemetryChatClient below FICC.
  • Updated WithDefaultAgentMiddleware to always place the deferred slot as the innermost decorator (directly above the leaf chat client).
  • Updated OpenTelemetryAgent to activate the deferred slot at construction time (and removed the previous per-run ChatClientFactory wrapping), plus added/adjusted unit tests including coverage for execute_tool spans.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
dotnet/src/Microsoft.Agents.AI/OpenTelemetryAgent.cs Activates the below-FICC OTel slot once during construction; removes per-run wiring so options flow through unchanged.
dotnet/src/Microsoft.Agents.AI/ChatClient/ChatClientExtensions.cs Ensures the deferred OTel slot is always present as the innermost decorator in the default agent chat-client pipeline.
dotnet/src/Microsoft.Agents.AI/ChatClient/DeferredOpenTelemetryChatClient.cs New inert/activatable decorator that swaps its target to an OpenTelemetryChatClient wrapper when activated.
dotnet/tests/Microsoft.Agents.AI.UnitTests/OpenTelemetryAgentTests.cs Updates expectations around options passing and adds tests proving execute_tool spans are emitted + slot activation behavior.

Comment thread dotnet/tests/Microsoft.Agents.AI.UnitTests/OpenTelemetryAgentTests.cs Outdated
…InvokingChatClient

OpenTelemetryAgent auto-wired OpenTelemetryChatClient above FICC, producing
OTel(FICC(leaf)). FICC resolved its ActivitySource at construction time as null,
so execute_tool spans were never emitted for tool-calling agents.

This repositions OTel below FICC, producing FICC(OTel(leaf)), via a deferred
NoOp slot pre-placed as the innermost decorator in WithDefaultAgentMiddleware
and activated once at the agent level.

- Add internal DeferredOpenTelemetryChatClient: inert DelegatingChatClient whose
  Activate(sourceName) swaps its target to inner.AsBuilder().UseOpenTelemetry().Build().
- WithDefaultAgentMiddleware always registers the slot innermost so it lands below FICC.
- OpenTelemetryAgent activates the slot once in its constructor and forwards run
  options straight through, removing the per-run ChatClientFactory outer wrap.
- Add and update unit tests, including a proof that execute_tool spans are emitted
  on the agent source and parented under invoke_agent.
@rogerbarreto rogerbarreto force-pushed the rogerbarreto/otel-below-ficc-tool-spans branch from 5d25408 to d26fe0c Compare June 22, 2026 17:16
@rogerbarreto rogerbarreto self-assigned this Jun 22, 2026
@rogerbarreto rogerbarreto added this pull request to the merge queue Jun 22, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 22, 2026
@peibekwe peibekwe added this pull request to the merge queue Jun 22, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 22, 2026
@peibekwe peibekwe added this pull request to the merge queue Jun 22, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 22, 2026
@rogerbarreto rogerbarreto added this pull request to the merge queue Jun 22, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 22, 2026
@peibekwe peibekwe merged commit 0979153 into microsoft:main Jun 22, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

.NET Usage: [Issues, PRs], Target: .Net

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: .NET: [Bug]: execute_tool spans missing when auto wiring OpenTelemetry via agent UseOpenTelemetry

5 participants