Skip to content

Record container images in test-metadata.json and fix embedding vllm-bench path for RHAIIS images #196

Description

@maryamtahhan

Problem:

  • Embedding test-metadata.json only records vllm_version from the server health check, not the container image tag. This made it impossible to tell from results alone whether RHAIIS 3.5.0 GA vs RC was used.
  • LLM benchmarks already record vllm_container_image and guidellm_container_image; embedding, audio, and MTEB do not.
  • When VLLM_BENCH_CONTAINER_IMAGE is set to a RHAIIS image, embedding benchmarks fail with crun: executable file /opt/venv/bin/vllm not found because baseline.yml / latency.yml hardcode the upstream image path.

Proposed changes:

  1. Add vllm_container_image (from effective_container_image) to embedding, audio, and MTEB metadata
  2. Add loadgen_container_image where a separate loadgen container is used
  3. Standardize field names across all playbooks (match llm-benchmark-auto.yml)
  4. Fix embedding bench command to use --entrypoint vllm instead of /opt/venv/bin/vllm
  5. Update dashboard/conversion tooling to display the new fields

Context: Successful RHAIIS 3.5.0 embedding runs used split images: RHAIIS server + upstream vllm-openai-cpu:v0.25.1 bench client. The reported error occurs when RHAIIS is used for both.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions