Question: distinguishing agent-reported completion from system-returned proof

[productmakerjason]

Hi Phoenix team.

quick agent observability/eval question.

I’m testing a small reliability boundary.

In my runs:

• black-box agents could often prepare a task or stop safely, but failed before receipt due to fetch/POST runtime limits
• a local POST-capable mini agent completed the same flow and received a receipt with receipt_id, proof_origin, and validation.status=passed

Question:
In Phoenix-style traces/evals, would you distinguish these states?

• agent-reported completion
• attempted execution without receipt
• system-returned receipt / durable confirmation

The failure mode I’m trying to capture is:
an agent may appear to have executed the right steps, but the target system never accepted or confirmed the task.

Is that distinction something Phoenix users already model, or would it usually be left to the app/test harness?

[dosubot[bot]]

Hi @productmakerjason! I'm Dosu and I’m helping the phoenix team.

Great question. Phoenix doesn't have a built-in semantic distinction between "agent-reported completion" vs "system-returned receipt/durable confirmation" — but it gives you the primitives to model it yourself.

What Phoenix provides out of the box:

• A hierarchical span structure: AGENT spans contain TOOL spans, so you can see whether a tool call the agent decided to make was actually executed [1]
• Tool call correlation: the agent's output includes a tool_call.id, and the corresponding tool result comes back as a message with a matching message.tool_call_id [2]. If a tool_call.id exists in the agent's output but no corresponding tool result message appears, that's your "attempted execution without receipt" signal.
• Span status codes (OK / ERROR) on both agent and tool spans [3]

How to model your three states:

1. Agent-reported completion → the AGENT span closes with status OK and a final output.value [4]
2. Attempted execution without receipt → a TOOL span exists (or a tool_call.id was emitted) but the span errored or never received a confirming result message
3. System-returned receipt / durable confirmation → you'd capture this as custom metadata on the tool span (e.g., receipt_id, proof_origin, validation.status) using Phoenix's metadata attribute support [5], or as the tool span's output.value

For the failure mode you described (agent appears to succeed but the target system never confirmed), the recommended approach is a custom evaluator that runs post-hoc over your traces. Phoenix's eval framework lets you export spans, apply code-based or LLM-based checks, and log results back as span annotations [6] [7]. You could write an evaluator that:

• Extracts all tool calls from agent output
• Verifies a corresponding tool span completed with OK status
• Checks that the tool output contains your expected receipt fields

So to answer directly: this distinction is not modeled by Phoenix conventions today, but it's absolutely something you can build on top of Phoenix's tracing + eval framework. Most users leave the "was this truly confirmed?" check to the app/test harness and then surface it in Phoenix via custom metadata or evaluators.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.