Skip to content

vllm parameter passthrough for stop sequences#4754

Closed
kdubovikov wants to merge 2 commits intohuggingface:mainfrom
kdubovikov:vllm-stop-sequences
Closed

vllm parameter passthrough for stop sequences#4754
kdubovikov wants to merge 2 commits intohuggingface:mainfrom
kdubovikov:vllm-stop-sequences

Conversation

@kdubovikov
Copy link
Contributor

What does this PR do?

I have added a passthrough for vllm stop sequence parameters. This can be very useful when implementing custom rollout functions. For example, for multi-step rollouts with intermediate tool calls.

@qgallouedec
Copy link
Member

thanks, but I think it's already supported via the argument generation_kwargs:

self.client.generate(
    prompts,
    max_tokens=4,
    temperature=0.0,
    generation_kwargs=dict(
        stop=["\n"],
        stop_token_ids=[stop_token_id],
        include_stop_str_in_output=False,
    ),
)

unless it's really needed I'd try to avoid adding new parameters to the client/server

@kdubovikov kdubovikov closed this Jan 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants