[llm][kv][8/N] Move tokenization in-process of LLMRouter ingress replica

[jeffreywang88]

Description

After this PR, pre-routing tokenization for KV-aware routing runs in-process inside LLMRouter, using the engine's own vLLM OnlineRenderer, instead of issuing a per-request RPC to a model replica's /tokenize.

KV-aware routing needs prompt token IDs to score replicas. Prior to this PR, the router obtained them via a per-request RPC to a replica's /tokenize. That round-trip sits directly on the routing critical path and contends with inference traffic, so it collapses under concurrency. Tokenizing in-process removes the RPC entirely, so the router keeps up as concurrency scales.

This PR also ships the enablement of fastokens:

llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="qwen3-0.6b",
        model_source="Qwen/Qwen3-0.6B",
    ),
    runtime_env=dict(
        env_vars={
            "VLLM_USE_FASTOKENS": "1",
        },
    ),
    deployment_config=dict(
        autoscaling_config=dict(min_replicas=1, max_replicas=2),
        request_router_config=dict(request_router_class=KVAwareRouter),
    ),
)

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request refactors the pre-routing tokenization mechanism by replacing remote /tokenize endpoint calls with an in-process Tokenizer that utilizes vLLM's OnlineRenderer directly. This keeps the router engine-agnostic by lazily importing the tokenizer only when KV-aware routing is enabled. Feedback on the changes suggests adding a guard when dumping tools in _render_chat to handle cases where tools might be raw dictionaries instead of Pydantic models, preventing potential AttributeErrors.

[gemini-code-assist]

If request.tools contains raw dictionaries (e.g., if validation allowed them or if they were bypassed), calling tool.model_dump() will raise an AttributeError. Using a guard like hasattr(tool, "model_dump") ensures compatibility with both Pydantic models and raw dictionaries.

        tool_dicts = (
            None
            if request.tools is None
            else [
                tool.model_dump() if hasattr(tool, "model_dump") else tool
                for tool in request.tools
            ]
        )

[gemini-code-assist]

Code Review

This pull request refactors the pre-routing tokenization mechanism to run in-process using vLLM's OnlineRenderer instead of making remote calls to the replica's /tokenize endpoint. It introduces lazy loading of the tokenizer for KV-aware routers to keep the non-KV ingress path engine-agnostic, and adds comprehensive exactness tests comparing the in-process tokenizer with Hugging Face's transformers library. The review feedback suggests initializing the tokenizer synchronously on the main thread during startup to avoid potential thread-safety issues with asyncio.to_thread, and wrapping background tasks in _schedule with try-except blocks to prevent exceptions from failing silently.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5b7e662. Configure here.}

[kouroshHakha]

does this approach work with fasttokens as well? https://github.com/vllm-project/vllm/blob/0b6aa3c47ce69a1f3c8a19cafe3b9dc2871d1f6b/docs/configuration/optimization.md?plain=1#L281 can we test that and confirm?

[jeffreywang88]

does this approach work with fasttokens as well? https://github.com/vllm-project/vllm/blob/0b6aa3c47ce69a1f3c8a19cafe3b9dc2871d1f6b/docs/configuration/optimization.md?plain=1#L281 can we test that and confirm?

Plumbed env var to enable that in fc63345 and added a test accordingly. Here's the API shape:

llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="qwen3-0.6b",
        model_source="Qwen/Qwen3-0.6B",
    ),
    runtime_env=dict(
        env_vars={
            "VLLM_USE_FASTOKENS": "1",
        },
    ),
    deployment_config=dict(
        autoscaling_config=dict(min_replicas=1, max_replicas=2),
        request_router_config=dict(request_router_class=KVAwareRouter),
    ),
)