[data][llm] Profile the vLLM engine in data LLM release benchmarks

[jeffreywang-anyscale]

Description

Goal: Help developers identify where the regression originates from in Ray Data LLM jobs.

Approach: In data LLM's release tests, we run benchmark with single-GPU generation workload. When there's a regression, it's not immediately clear if the regression originates from the vLLM engine or Ray Data LLM. Once #60385 lands, we should be able to profile the vLLM engine (e.g. TPOT, token throughput, E2E request latency) and print the metrics in the benchmark output so that it's clear whether the regression results from the vLLM engine. We can also profile the ray data LLM job.

Relevant file: https://github.com/ray-project/ray/blob/ad1b87448fec4db7ef11f1697f9bc02ae6a7ba09/release/llm_tests/batch/test_batch_single_node_vllm.py

Hardware requirement: You will need a GPU for this task.

Use case

No response

[Nishant-ZFYII]

Hi @jeffreywang-anyscale — I'd like to take this on.
My plan:

1. Wait for [Data][LLM] Add vLLM metrics export and Data LLM Grafana dashboard #60385 to land, then pull the updated test_batch_single_node_vllm.py
2. Add vLLM engine profiling metrics (TPOT, token throughput, E2E request latency) using vLLM's built-in metrics interface
3. Add Ray Data LLM job-level profiling to separate Ray overhead from engine overhead
4. Print both metric sets in the benchmark output for clear regression attribution

I have GPU . I'm an active contributor to OpenVINO (1 merged PR, 9 in review) working on inference optimization — similar profiling/benchmarking domain.
Happy to start with a draft PR once #60385 merges. Any preferences on the metrics format or output structure?

[jeffreywang-anyscale]

Hey @Nishant-ZFYII thank you for volunteering! #60385 is merged, feel free to go ahead with your plan.

Any preferences on the metrics format or output structure?

We can print them to stdout and persist them in JSON files in a cloud storage so that we can compare them across runs.

[Nishant-ZFYII]

Thanks for the go ahead, will start working on it soon.