Single Agent Tool LlamaIndex Instrumentation #86

shuningc · 2025-11-26T23:41:45Z

Adding a separate workflow agent instrumentation mechanism with a wrapper function:
✅ Dual instrumentation paths for backward compatibility
✅ Non-invasive: Uses monkey-patching, no code changes needed
✅ Async-aware: Background task doesn't block user code
✅ Context propagation: Manual for workflows, automatic for callbacks
✅ Shared TelemetryHandler: Both paths use same handler instance

User code:
agent = ReActAgent(tools=[...])
handler = agent.run("Calculate 5*3+2")
↓
↓ (intercepted by wrap_agent_run)
↓

wrap_agent_run():
Line 137: handler = wrapped(*args, **kwargs)
↓
└─→ Calls original ReActAgent.run()
↓
└─→ Returns WorkflowHandler
(contains stream of events)

Line 159: await instrumentor.instrument_workflow_handler(handler, ...)
↓
└─→ Pass handler to instrumentor

Line 169: return handler # Return to user
↓
└─→ User can await it

instrument_workflow_handler(workflow_handler):
Line 43: async for event in workflow_handler.stream_events():
↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
THIS IS THE CONSUMPTION!

       We're iterating through the event stream
       that the WorkflowHandler provides

Below is a sample trace when running this app locally
https://shw-playground.signalfx.com/#/apm/traces/f37aebbad8b26e16fd20810c5b23a017?tab=waterfall

- Add embedding event handlers (_handle_embedding_start, _handle_embedding_end) - Extract model name, input texts, and dimension count from embedding events - Create vendor_detection.py module with VendorRule-based provider detection - Support 13+ embedding providers (OpenAI, Azure, AWS, Google, Cohere, etc.) - Add test_embedding_instrumentation.py with single and batch embedding tests - Update README with embedding documentation and provider list - Tested successfully with OpenAI embeddings API

JWinermaSplunk · 2025-12-09T21:07:14Z

...-instrumentation-llamaindex/src/opentelemetry/instrumentation/llamaindex/callback_handler.py

+
+        # Get the currently active span to establish parent-child relationship
+        # First try to get from active agent context (workflow-based agents)
+        parent_span = None


Might be a good candidate for a helper function? i.e. 154-164

Created a helper function and refactored.

Changed start_agent_invocation -> start_agent and stop_agent_invocation -> stop_agent to match the actual TelemetryHandler API. These methods are called when CBEventType.AGENT_STEP events are emitted (rare in modern LlamaIndex, but used by legacy AgentRunner and custom workflows).

adityamehra · 2026-01-22T20:14:58Z

...-instrumentation-llamaindex/src/opentelemetry/instrumentation/llamaindex/callback_handler.py

+        if not self._handler:
+            return None
+        current_id = parent_id
+        while current_id:


It might be better to do some defensive programming and use a setto track the visited invocation similar to what we do in opentelemetry.instrumentation.langchain.callback_handler.LangchainCallbackHandler._find_nearest_agent

Added visited checking

adityamehra · 2026-01-22T20:36:00Z

...-instrumentation-llamaindex/src/opentelemetry/instrumentation/llamaindex/callback_handler.py

            request_presence_penalty=presence_penalty,
            request_stop_sequences=stop if stop else [],
            request_seed=seed,
+            parent_run_id=parent_id if parent_id else None,


I don't think we plan on passing parent_run_id from the instrumentation layer for the types. This needs to go the invocation_manager.py

cc: @keith-decker @wrisa

adityamehra · 2026-01-22T20:38:57Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+    """
+    Wrap agent.run() to instrument workflow events.
+
+    This creates a root agent span immediately when agent.run() is called,


Why are we NOT creating a root workflow span?

At root level (no parent):

If [_is_agent_root()] detects agent markers → creates AgentInvocation
Otherwise → creates Workflow (generic chain/sequence)

This is agent call.
In the RAG instrumentation, I created the workflow span

adityamehra · 2026-01-22T20:40:46Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+        # Create root agent invocation before workflow starts
+        root_agent = AgentInvocation(
+            name=f"agent.{type(instance).__name__}",
+            run_id=str(uuid4()),


There's no need to pass run_id when creating new types as it is a default field. Also, we shouldn't be passing run_id or parent_run_id from the instrumentation layer

adityamehra · 2026-01-22T20:42:07Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+        from uuid import uuid4
+
+        # Create root agent invocation before workflow starts
+        root_agent = AgentInvocation(


I might be missing something but it's better to create a root workflow span and make this root_agent it's child

The design logic here is
For root-level calls (parent_run_id is None):

Check if it's an agent using [_is_agent_root()] - looks for agent tags/metadata
If agent → create AgentInvocation span
If not agent → create Workflow span (for chains, sequences, etc.)

Here is the langchain example: https://github.com/signalfx/splunk-otel-python-contrib/blob/main/instrumentation-genai/opentelemetry-instrumentation-langchain/src/opentelemetry/instrumentation/langchain/callback_handler.py#L232

Added workflow span

…alfx/splunk-otel-python-contrib into singleAgent-tool-llamaindex

adityamehra · 2026-01-22T20:44:17Z

...on-genai/opentelemetry-instrumentation-llamaindex/examples/travel_planner_k8s/main_server.py

+
+
+# OAuth2 Token Manager for CircuIT
+class OAuth2TokenManager:


Move this to a util dir and share with zero-code example

adityamehra · 2026-01-26T04:35:08Z

instrumentation-genai/opentelemetry-instrumentation-llamaindex/tests/test_workflow_agent.py

+    return a + b
+
+
+def setup_telemetry():


This method is not used.

adityamehra · 2026-01-26T04:36:14Z

instrumentation-genai/opentelemetry-instrumentation-llamaindex/tests/test_workflow_agent.py

+    return tracer_provider
+
+
+def setup_telemetry_with_memory():


How does having an InMemorySpanExporter help with testing vs the Console based exporters?

Enables assertions - you can verify span count, names, attributes, hierarchy without printing out the spans

adityamehra · 2026-01-26T04:43:16Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+    Wrap agent.run() to instrument workflow events.
+
+    This creates a Workflow span as the root (since ReActAgent inherits from Workflow),
+    then creates an AgentInvocation span nested inside it. This matches the CrewAI


nit: Let's remove reference to CrewAI

adityamehra · 2026-01-26T04:45:02Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+            initial_input=str(user_msg),
+            attributes={},
+        )
+        root_workflow.framework = "llamaindex"


We are already passing "llamaindex" on line 120 when instantiating the Workflow(). This looks redundant

adityamehra · 2026-01-26T04:53:47Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+        self._active_tools = {}  # tool_id -> ToolCall
+        self._root_agent = None  # Reference to the root agent span
+
+    async def instrument_workflow_handler(


Does llamaindex not support synchronous workflows?

LlamaIndex's Workflow-based agents are fundamentally async-only. The legacy one is synchronous but the newer ReactAgent(workflow-based agents) are async only.

adityamehra · 2026-01-26T06:02:54Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+
+    # Get TelemetryHandler from callback handler if available
+    from llama_index.core import Settings
+    from opentelemetry import context


Avoid context management on the instrumentation layer. Pls use genai utils for context management

adityamehra · 2026-01-26T06:12:29Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+    parent_context = None
+    if telemetry_handler:
+        # Level 1: Create root workflow span (since ReActAgent inherits from Workflow)
+        root_workflow = Workflow(


Looking at the code flow I think this will wrap every agent call inside a new workflow span but we need a single workflow root span to which agent span are attached, e.g.

- workflow - agent 1 - agent 2 - agent 3

adityamehra · 2026-01-26T06:12:54Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+    # Call the original run() method to get the workflow handler
+    handler = wrapped(*args, **kwargs)
+
+    if telemetry_handler and root_workflow and root_agent and parent_context:


remove parent_context from here

adityamehra · 2026-01-26T06:22:13Z

...entation-llamaindex/src/opentelemetry/instrumentation/llamaindex/workflow_instrumentation.py

+    def __init__(self, handler: TelemetryHandler):
+        self._handler = handler
+        self._active_tools = {}  # tool_id -> ToolCall
+        self._root_agent = None  # Reference to the root agent span


LLamaIndex must have multi-agent support. I might be wrong but pls check. How will we track the multi-agent apps?

Found this in the docs - https://developers.llamaindex.ai/python/framework/understanding/agent/multi_agent/

Yes, I actually dealt with multi-agent scenario in the code. The PR title is confusing, i will change it.

zhirafovod · 2026-01-26T18:20:37Z

@shuningc Rerun the llamaindex demo app and capture/attach screenshots and links to these screenshots in O11y.
Capture the metrics and screenshots ensuring that gen_ai.framework is set to llamaindex for the metrics

shuningc added 4 commits November 25, 2025 15:35

Adding Llamaindex llm instrumentation spike

d411b68

Updating readme

79fd8aa

Adding single agent tool instrumentation for llamaindex

43cfe8d

shuningc requested review from a team as code owners November 26, 2025 23:41

shuningc added 4 commits December 8, 2025 00:57

Merge branch 'main' into singleAgent-tool-llamaindex

4d85db1

Adding travel_planner_agent example to llamaindex instrumentation

1cabb4d

Fix ruff linting errors: remove unused imports and variables

4765dcb

Apply ruff formatting to LlamaIndex instrumentation files

8e3b049

shuningc changed the title ~~DRAFT Single Agent Tool LlamaIndex Instrumentation~~ Single Agent Tool LlamaIndex Instrumentation Dec 9, 2025

JWinermaSplunk reviewed Dec 9, 2025

View reviewed changes

shuningc added 17 commits December 12, 2025 08:38

Adding deployment files for demo tarvel-planner app

ddbb103

Merge branch 'main' into singleAgent-tool-llamaindex

8458ff6

Merge branch 'main' into singleAgent-tool-llamaindex

b1c1ca6

Creating helper function and refactoring

b771249

Fixing ruff format errors

867fb72

Adding circuit api changes for demo app

d84dac9

Fixing linting errors

b59d1e6

Merge branch 'main' into singleAgent-tool-llamaindex

116289d

Merge branch 'main' into singleAgent-tool-llamaindex

fb56463

Clear toolcall instrumentation logic

af748a6

Adding linting fix

cc2d782

Updating travel-planner app deployment

ca29f90

Linting fix

21df4b0

Updating agentinvocation to emit agent duration metric

3917194

Adding zero-code demo app

2725ab3

Bumping image version for multi-agent-travel-planner

b198de2

adityamehra requested changes Jan 22, 2026

View reviewed changes

shuningc and others added 7 commits January 23, 2026 05:27

Fixing callback_handler based on comments

e407618

Merge branch 'main' into singleAgent-tool-llamaindex

5ddd20b

Linting fix

9f3e79e

lint

523414e

Dealing with some comments for workflow span

ddf69f6

Merge branch 'main' into singleAgent-tool-llamaindex

1e3a753

Merge branch 'singleAgent-tool-llamaindex' of https://github.com/sign…

7c0a99b

…alfx/splunk-otel-python-contrib into singleAgent-tool-llamaindex

adityamehra requested changes Jan 26, 2026

View reviewed changes

shuningc added 3 commits January 26, 2026 13:08

Dealing with comments

7a168b2

Merge branch 'main' into singleAgent-tool-llamaindex

bd96dca

Adding function agent, codeact agent, multiagent workflow to the wrapper

1fc07a5

Single Agent Tool LlamaIndex Instrumentation #86

Are you sure you want to change the base?

Single Agent Tool LlamaIndex Instrumentation #86

Uh oh!

Conversation

shuningc commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhirafovod commented Jan 26, 2026

Uh oh!

shuningc commented Nov 26, 2025 •

edited

Loading