Skip to content

[Performance]: Improve swap_states by swapping active token prefixes #34731

@pjo256

Description

@pjo256

Proposal to improve performance

InputBatch.swap_states() in vllm/v1/worker/gpu_input_batch.py currently does expensive full-row token buffer
swapping on the request reordering path. This can be reduced by swapping only the active token prefix.

cc @LucasWilkinson since the original TODO in #14253 calls out optimizing this copy path.

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    performancePerformance-related issues

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions