[Bug]: /v1/responses: Protocol drift and malformed tool aggregation breaking official OpenAI SDK compatibility

[scott8]

Your current environment

compile_project_v2.py

The output of python collect_env.py

root@spark:/workspace/vllm# python3 collect_env.py
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (aarch64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.12.0.dev20260406+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar  3 2026, 12:15:18) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.17.0-1014-nvidia-aarch64-with-glibc2.39

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.2.51
CUDA_MODULE_LOADING set to   :
GPU models and configuration : GPU 0: NVIDIA GB10
Nvidia driver version        : 580.142
cuDNN version                : Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.9.20.0
/usr/lib/aarch64-linux-gnu/libcudnn_adv.so.9.20.0
/usr/lib/aarch64-linux-gnu/libcudnn_cnn.so.9.20.0
/usr/lib/aarch64-linux-gnu/libcudnn_engines_precompiled.so.9.20.0
/usr/lib/aarch64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.20.0
/usr/lib/aarch64-linux-gnu/libcudnn_graph.so.9.20.0
/usr/lib/aarch64-linux-gnu/libcudnn_heuristic.so.9.20.0
/usr/lib/aarch64-linux-gnu/libcudnn_ops.so.9.20.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  20
On-line CPU(s) list:                     0-19
Vendor ID:                               ARM
Model name:                              Cortex-X925
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      10
Socket(s):                               1
Stepping:                                r0p1
Frequency boost:                         disabled
CPU(s) scaling MHz:                      100%
CPU max MHz:                             3900.0000
CPU min MHz:                             1378.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
Model name:                              Cortex-A725
Model:                                   1
Thread(s) per core:                      1
Core(s) per socket:                      10
Socket(s):                               1
Stepping:                                r0p1
CPU(s) scaling MHz:                      100%
CPU max MHz:                             2808.0000
CPU min MHz:                             338.0000
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti ecv afp wfxt
L1d cache:                               1.3 MiB (20 instances)
L1i cache:                               1.3 MiB (20 instances)
L2 cache:                                25 MiB (20 instances)
L3 cache:                                24 MiB (2 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-19
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.7
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.20.0.48
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.29.7
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.12.0.dev20260406+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0.dev20260402+cu130
[pip3] torchvision==0.27.0.dev20260406+cu130
[pip3] transformers==5.5.0
[pip3] triton==3.7.0+git9c288bc5
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.19.1.dev0+g2a69949bd.d20260408 (git sha: 2a69949bd, date: 20260408)
vLLM Build Flags:
  CUDA Archs: 12.1a; ROCm: Disabled
GPU Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-19    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=13.2 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=580,driver<581 brand=grid,driver>=580,driver<581 brand=tesla,driver>=580,driver<581 brand=nvidia,driver>=580,driver<581 brand=quadro,driver>=580,driver<581 brand=quadrortx,driver>=580,driver<581 brand=nvidiartx,driver>=580,driver<581 brand=vapps,driver>=580,driver<581 brand=vpc,driver>=580,driver<581 brand=vcs,driver>=580,driver<581 brand=vws,driver>=580,driver<581 brand=cloudgaming,driver>=580,driver<581 brand=unknown,driver>=590,driver<591 brand=grid,driver>=590,driver<591 brand=tesla,driver>=590,driver<591 brand=nvidia,driver>=590,driver<591 brand=quadro,driver>=590,driver<591 brand=quadrortx,driver>=590,driver<591 brand=nvidiartx,driver>=590,driver<591 brand=vapps,driver>=590,driver<591 brand=vpc,driver>=590,driver<591 brand=vcs,driver>=590,driver<591 brand=vws,driver>=590,driver<591 brand=cloudgaming,driver>=590,driver<591
TORCH_CUDA_ARCH_LIST=12.1a
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=13.2.0
MAX_JOBS=16
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

🐛 Describe the bug

Here’s what appears to be some legitimate vLLM errors. I have a custom AI client using the Python openai library. Gemini 3 Flash (Preview) generated this after working with me on my client implementation. The context for finding this is that we are trying to make my client work with the official OpenAI library and vLLM’s /v1/responses endpoint. We currently have to use a lower-level manual SSE parser to handle the stream because vLLM’s output is incompatible with the SDK’s internal assertions.

Bug Report: vLLM /v1/responses Protocol Violations Breaking Official OpenAI SDK
Environment:

• Server: vLLM version 0.19.1.dev0+g2a69949bd.d20260408
• Model: /models/qwen35-122b-hybrid (Reasoning enabled)
• Client: official openai-python SDK (v 2.31.0, Python 3.14.2)
• Primary Reference: https://community.openai.com/t/responses-api-streaming-the-simple-guide-to-events/1363122 (Referred to as the “OpenAI Field Guide” below)
• vLLM Protocol Reference: https://docs.vllm.ai/en/latest/api/vllm/entrypoints/openai/responses/protocol/
• Supplementary Reference: https://learn.microsoft.com/en-us/python/api/azure-ai-agentserver-core/azure.ai.agentserver.core.models.openai.responsereasoningtextdeltaevent?view=azure-python-preview (Confirms reasoning_text.delta standard)

1. CRITICAL: Malformed JSON Aggregation in Tool Calls
   Issue: In turns requiring multiple simultaneous tool calls, vLLM incorrectly concatenates JSON argument strings in the .done event instead of scoping them to their specific output_index. This causes the official openai-python SDK to crash with an AssertionError.

Reproduction Proof (Raw SSE Trace):

Tool Call 1 started (output_index: 2)
Tool Call 2 started (output_index: 3)
Raw arguments field content in .done event:
{"file_path": "foo.txt", "reason": "..."}{"file_path": "bar.txt", "reason": "..."}
Violation: As per Table E of the OpenAI Field Guide, the arguments field in a function_call_arguments.done event must be a single, valid JSON object. vLLM is producing a }{ concatenation, which is invalid JSON.

Impact: The official SDK’s high-level parser (client.responses.stream()) validates the JSON buffer during assembly. Encountering concatenated objects causes an internal state mismatch and an immediate crash.

2. PERFORMANCE: Redundant Instruction Echoing (Token Waste)
   Issue: vLLM echoes the full instructions block (system prompt) and all request parameters in multiple SSE events at the start of every stream.

Reproduction Proof:

• Instruction Size: 1,111 characters
• Event 1 (response.created): Instructions echoed back (1,111 chars)
• Event 2 (response.in_progress): Instructions echoed back again (1,111 chars)
• Total Redundant Characters: 2,222 (Double the prompt size).

The Impact: This violates the efficiency principles of the Responses API. In applications with large system prompts, this double-echoing induces significant network latency and bandwidth waste. Standard behavior should only echo parameters once (typically in response.created).

3. COMPATIBILITY: Non-Standard Reasoning Lifecycle Events
   Issue: vLLM uses custom event types response.reasoning_part.added and response.reasoning_part.done.

Violation: While reasoning_text.delta is a recognized standard, the lifecycle events for these parts should follow the standard response.content_part.added (type: reasoning_text) schema defined in the OpenAI Field Guide (Table D).

Impact: Invented event names like reasoning_part cause drift from the official SDK’s event-dispatch registry and trigger internal assertions during stream parsing.

Suggested Remediation:
Fix Aggregator: Ensure function_call_arguments strings contain only the JSON for the specific output_index being finalized. Do not concatenate.

Align SSE Schema: Transition reasoning_part lifecycle events to the standard content_part schema.

Optimize Network Traffic: Suppress the redundant instructions echo in events following the initial response.created.

Attached is a script to reproduce these problems. Please let me know if you need more information.

reproduce_vllm_bugs.py

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ianliuy]

I traced through the full /v1/responses streaming code path for all three reported issues. Here's what I found:

TL;DR

Sub-issue	Verdict
1. Malformed tool call JSON aggregation	Confirmed bug root cause identified
2. Redundant instruction echoing	Not a bug this is per OpenAI spec
3. Non-standard reasoning events	Known issue intentional workaround (see below)

---

1. Tool Call JSON Aggregation (Confirmed Bug)

What's happening?

When the model makes multiple simultaneous tool calls, the function_call_arguments.done event contains all tool calls' arguments concatenated together, producing invalid JSON like:

{"file_path": "foo.txt", "reason": "..."}{"file_path": "bar.txt", "reason": "..."}

Root cause

This only affects the non-Harmony streaming path (_process_simple_streaming_events() in serving.py). Two defects:

1. previous_delta_messages is never reset between tool calls. Every argument delta from every tool call accumulates in the same list. At finalization (around line 1690), all deltas are "".join()-ed into a single string producing the }{ concatenation.

2. Tool-call-to-tool-call transition emits wrong event types. When a second tool call starts, the code emits response.output_text.done and response.content_part.done (which are for text output), instead of response.function_call_arguments.done for the previous tool call.

The Harmony path handles this correctly via emit_previous_item_done_events() + state.reset_for_new_item(), which properly finalizes each tool call before starting the next.

Fix approach

In the tool-call transition block:

• Emit function_call_arguments.done for the previous tool call with only its accumulated arguments
• Emit output_item.done with the correct function_call type (not message)
• Reset previous_delta_messages so the next tool call starts clean
• Remove the incorrect text/content done events from the transition

---

2. Redundant Instruction Echoing (Not a Bug)

The OpenAI SDK types confirm that response.created, response.in_progress, and response.completed all carry a full Response object which includes instructions, tools, metadata, etc. This is the specified event structure, not redundant echoing. vLLM is correctly following the spec here.

---

3. Non-Standard Reasoning Events (Known Workaround)

vLLM uses custom response.reasoning_part.added / response.reasoning_part.done events. The OpenAI spec uses response.content_part.added / response.content_part.done with part.type = "reasoning_text" instead.

However, the vLLM codebase already has TODO comments acknowledging this:

# TODO: this code can be removed once
# https://github.com/openai/openai-python/issues/2634 has been resolved

This was an intentional workaround for an upstream SDK issue, not an oversight.

---

I'd like to submit a fix for sub-issue 1 (the tool call JSON aggregation bug). Happy to address sub-issue 3 separately once the upstream SDK issue is resolved.

[ianliuy]

Fix for Sub-issue 1: Malformed Tool Call JSON Aggregation

What is the bug?

When a model makes multiple tool calls in a single response (e.g., "check the weather in NYC and London"), the function_call_arguments.done SSE event contains all tool calls' arguments smashed together:

{"city":"NYC"}{"city":"London"}

That's not valid JSON. The official OpenAI Python SDK tries to parse it, hits a wall, and crashes.

Who is affected?

Anyone using vLLM's /v1/responses endpoint with streaming + a model/prompt that triggers 2+ parallel tool calls. Single tool calls work fine. The Harmony path is also unaffected this only hits the non-Harmony (simple) streaming path.

Where is the bug in the code?

In _process_simple_streaming_events() in serving.py. The method uses a list called previous_delta_messages to accumulate argument fragments as they stream in. The problem: this list is never cleared between tool calls.

So when tool call #1 finishes and tool call #2 starts, all of #1's argument fragments are still sitting in the list. At the end of the stream, everything gets "".join()-ed into one big string.

On top of that, the transition from tool call #1 #2 emits the wrong event types entirely output_text.done and content_part.done (which are for text output) instead of function_call_arguments.done and output_item.done with a ResponseFunctionToolCall.

Why does the Harmony path work?

The Harmony path has a proper state machine (StreamingState) with reset_for_new_item() that clears state between items, and emit_function_call_done_events() that correctly finalizes each tool call individually.

How did I fix it?

Minimal surgical change I added a check at the tool-call transition point: "was the previous item also a tool call?"

• Tool Tool transition: Emit the correct done events (function_call_arguments.done + output_item.done with ResponseFunctionToolCall) for the outgoing tool call, then reset previous_delta_messages = [] before setting up the new one.
• Text Tool transition: Keep the existing behavior unchanged.

This means each tool call's argument fragments stay isolated. The finalization block at the end of the stream now only sees the last tool call's deltas, so "".join() naturally produces valid single-tool-call JSON.

Bonus: Also fixed a field name bug in the Harmony path's emit_function_call_done_events() it was passing item_id= instead of id= to ResponseFunctionToolCall. Because pydantic has extra="allow", this silently created a junk item_id field while the real id stayed None.

What does the correct event sequence look like?

Before the fix (2 tool calls):

output_item.added (tool 0)
  function_call_arguments.delta  N
  output_text.done           wrong event type
  content_part.done          wrong event type
  output_item.done (message)  wrong item type
output_item.added (tool 1)
  function_call_arguments.delta  N
  function_call_arguments.done (args = tool0 + tool1 concatenated)  corrupted
  output_item.done (function_call with concatenated args)  corrupted

After the fix:

output_item.added (tool 0)
  function_call_arguments.delta × N
  function_call_arguments.done (only tool 0 args)  correct
  output_item.done (function_call for tool 0)  correct
output_item.added (tool 1)
  function_call_arguments.delta  N
  function_call_arguments.done (only tool 1 args)  correct
  output_item.done (function_call for tool 1)  correct

What tests were added?

Test	Validates
test_tool_call_transition_emits_correct_done_events	Tooltool transition emits function_call_arguments.done, not output_text.done
test_multi_tool_call_arguments_not_concatenated	Each done event has its own valid JSON, not concatenated
test_multi_tool_call_distinct_output_indices	Each tool call gets a unique, increasing output_index
test_single_tool_call_unchanged	Regression: single tool call still works

PR incoming.

[chaunceyjiang]

I'll take a look at this issue.

[scott8]

openai/openai-python#263 is Closed.

[ianliuy]

@scott8 Heads up the issue referenced in vLLM's TODO comment for the reasoning events workaround is openai/openai-python#2634 ("Types for ResponseReasoningPartAdded, ResponseReasoningPartDone"), not #263. The one you linked (#263) is an unrelated 2023 issue about invalid URLs.

#2634 is still open it's a feature request from vLLM's own @qandrew asking the OpenAI Python SDK to add type definitions for ResponseReasoningPartAdded / ResponseReasoningPartDone. These event types are what GPT-OSS actually emits during streaming (you can see it in the raw SSE traces), but the official Python SDK doesn't have type definitions for them yet. So vLLM's custom events aren't "invented" they match GPT-OSS's real behavior; the SDK just hasn't caught up.

Once #2634 is resolved (either by adding new types or extending ResponseContentPartDone), the workaround in vLLM can be cleaned up.

[scott8]

Thanks for the feedback. My mistake. I made a copy and paste error. SMH.