feat(audio): batch Qwen3-ASR transcriptions

[qinxuye]

Summary

• batch concurrent Qwen3-ASR transcription requests through the model's native list inference
• preserve per-request language and response formatting while grouping compatible inference kwargs
• use a Qwen3-ASR-specific 100 ms aggregation window by default while keeping batch_interval configurable
• harden the shared batch worker against cancellation and result-count mismatches
• enforce the shared BatchMixin size limit while preserving FIFO order for requests deferred to the next batch

Why

Qwen3-ASR supports batched transcription, but Xinference currently invokes each concurrent audio request separately. This change lets compatible requests share one model call while preserving the existing API response behavior.

The shared 3 ms batching interval is too short for differently sized audio uploads to reach the model actor together. Qwen3-ASR therefore uses a 100 ms default without changing the global BatchMixin default.

BatchMixin is shared by embedding, rerank, and Qwen3-ASR models. Because one queued request can contain multiple inputs, checking only the current accumulated size is insufficient: the next request can push a batch past its configured limit. The worker now retains such a request as a FIFO-preserving pending item for the next batch. A single request that already exceeds the limit still runs alone because the mixin cannot split one request.

Validation

• pytest -q xinference/model/tests/test_batch.py xinference/model/audio/tests/test_qwen3_asr.py — 8 passed
• pre-commit run --files xinference/model/batch.py xinference/model/tests/test_batch.py xinference/model/audio/qwen3_asr.py — passed
• git diff --check — passed

The shared batch tests cover a request that would overflow the remaining capacity, exact-capacity batches, FIFO preservation across deferred requests, and a single oversized request.

GPU end-to-end results

Tested through a real Xinference service on an NVIDIA GeForce RTX 3090 Ti with Qwen3-ASR-1.7B, batch_size=2, and the default batch_interval=0.1.

Method:

• excluded one warm-up run
• ran three measured rounds and alternated sequential/batch order
• sequential time is the sum of the two individual request times
• batch time is wall-clock time until both concurrent requests complete
• compared the returned text for every input across every sequential and batch run

Inputs	Sequential sum, median	Batch wall time, median	Result
蝶恋花·春景.mp3 ×2 (180.024 s each)	3.232 s	2.033 s	37.1% lower wall time
synthesized_speech_92345053.mp3 ×2 (106.606 s each)	8.362 s	4.409 s	47.3% lower wall time
Mixed 180.024 s + 106.606 s inputs	5.877 s	6.649 s	13.1% higher wall time

Debug logs confirmed that sequential runs invoked native batch sizes [1, 1], while concurrent runs invoked [2].

Sequential and batched outputs matched exactly for both files in every measured round. This validates output parity, not transcript ground-truth accuracy.

The mixed-duration pair is slower because native batching pads work toward the longest item: the two inputs contain 286.630 seconds of audio in total, while a two-item batch can process approximately 360.048 seconds of padded frames. The throughput gain is therefore strongest when requests have similar lengths.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[qinxuye]

/gemini review

[gemini-code-assist]

Code Review

This pull request implements batching support for the Qwen3 ASR model by integrating BatchMixin and refactoring transcription methods, accompanied by new unit tests. The reviewer feedback suggests popping max_inference_batch_size from kwargs to avoid TypeErrors, handling None values in the language list to prevent transcription errors, and adding a close method to BatchMixin to clean up the background task and prevent memory leaks.

[OliverBryant]

Nice work. The batching integration follows the existing create_embedding @extensible/.batch convention, and the BatchMixin._process_batch hardening (moving the count check inside try, dropping cancelled futures, restartable processor task) is a solid robustness improvement. Tests cover ordering, batch-size, kwargs grouping, and cancellation survival. LGTM.

Left two minor, non-blocking notes below. The only one I'd genuinely suggest acting on is a mention in the PR description of the shared batch.py boundary change, since it affects every batch-capable model, not just Qwen3-ASR.