Proposal to improve performance
InputBatch.swap_states() in vllm/v1/worker/gpu_input_batch.py currently does expensive full-row token buffer
swapping on the request reordering path. This can be reduced by swapping only the active token prefix.
cc @LucasWilkinson since the original TODO in #14253 calls out optimizing this copy path.
Report of performance regression
No response
Misc discussion on performance
No response
Your current environment (if you think it is necessary)
The output of `python collect_env.py`
Before submitting a new issue...
Proposal to improve performance
InputBatch.swap_states()invllm/v1/worker/gpu_input_batch.pycurrently does expensive full-row token bufferswapping on the request reordering path. This can be reduced by swapping only the active token prefix.
cc @LucasWilkinson since the original TODO in #14253 calls out optimizing this copy path.
Report of performance regression
No response
Misc discussion on performance
No response
Your current environment (if you think it is necessary)
Before submitting a new issue...