Skip to content

Improve how we identify and run e2e smoke tests #2336

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dhuangnm wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Improve how we identify and run e2e smoke tests #2336

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7b1c1c9
Support smoke test group in config files to allow run e2e tests with …
dhuangnm Jan 15, 2026
a2d054d
remove rhaiis-e2e-smoke.list
dhuangnm Feb 5, 2026
3d82f58
fix a bug
dhuangnm Feb 5, 2026
ed81493
change group to rhaiis to be more clear
dhuangnm Feb 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion tests/e2e/vLLM/configs/fp4_nvfp4.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
cadence: "nightly"
test_type: "regression"
test_group: "rhaiis"
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
scheme: NVFP4
num_calibration_samples: 20
dataset_id: HuggingFaceH4/ultrachat_200k
dataset_split: train_sft
dataset_split: train_sft
3 changes: 2 additions & 1 deletion tests/e2e/vLLM/configs/fp8_dynamic_per_token.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
cadence: "nightly"
test_type: "regression"
test_group: "rhaiis"
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
scheme: FP8_DYNAMIC
scheme: FP8_DYNAMIC
3 changes: 2 additions & 1 deletion tests/e2e/vLLM/configs/int8_channel_weight_static_per_tensor_act.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
cadence: "nightly"
test_type: "regression"
test_group: "rhaiis"
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
recipe: tests/e2e/vLLM/recipes/INT8/recipe_int8_channel_weight_static_per_tensor_act.yaml
dataset_id: HuggingFaceH4/ultrachat_200k
dataset_split: train_sft
scheme: W8A8_channel_weight_static_per_tensor
scheme: W8A8_channel_weight_static_per_tensor
3 changes: 2 additions & 1 deletion tests/e2e/vLLM/configs/kv_cache_gptq_tinyllama.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
cadence: "nightly"
test_type: "regression"
test_group: "rhaiis"
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
recipe: tests/e2e/vLLM/recipes/kv_cache/gptq.yaml
dataset_id: HuggingFaceH4/ultrachat_200k
dataset_split: train_sft
scheme: kv_cache_default_gptq_tinyllama
scheme: kv_cache_default_gptq_tinyllama
3 changes: 2 additions & 1 deletion tests/e2e/vLLM/configs/sparse2of4_fp8_dynamic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
cadence: "nightly"
test_type: "regression"
test_group: "rhaiis"
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
recipe: tests/e2e/vLLM/recipes/Sparse_2of4/recipe_sparse_2of4_fp8_dynamic.yaml
scheme: sparse2of4_fp8_dynamic
dataset_id: HuggingFaceH4/ultrachat_200k
dataset_split: train_sft
dataset_split: train_sft
3 changes: 2 additions & 1 deletion tests/e2e/vLLM/configs/w4a16_actorder_weight.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
cadence: "nightly"
test_type: "regression"
test_group: "rhaiis"
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
recipe: tests/e2e/vLLM/recipes/actorder/recipe_w4a16_actorder_weight.yaml
dataset_id: openai/gsm8k
dataset_config: main
dataset_split: train
scheme: W4A16_actorder_weight
save_dir: TinyLlama-1.1B-Chat-v1.0-actorder-weight
save_dir: TinyLlama-1.1B-Chat-v1.0-actorder-weight
3 changes: 2 additions & 1 deletion tests/e2e/vLLM/configs/w4a16_grouped_quant_asym_awq.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
cadence: "nightly"
test_type: "regression"
test_group: "rhaiis"
model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
recipe: tests/e2e/vLLM/recipes/WNA16/recipe_w4a16_awq_asym.yaml
dataset_id: HuggingFaceH4/ultrachat_200k
dataset_split: train_sft
scheme: W4A16_weight_asym_awq
save_dir: TinyLlama-1.1B-Chat-v1.0-w4a16-asym-awq
gpu_memory_utilization: 0.85
gpu_memory_utilization: 0.85
7 changes: 0 additions & 7 deletions tests/e2e/vLLM/rhaiis-e2e-smoke.list
View file
Open in desktop

This file was deleted.

45 changes: 33 additions & 12 deletions tests/e2e/vLLM/run_tests_in_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,60 @@

SUCCESS=0

while getopts "c:t:" OPT; do
while getopts "c:t:g:" OPT; do
case ${OPT} in
c )
CONFIG="$OPTARG"
;;
t )
TEST="$OPTARG"
;;
g )
GROUP="$OPTARG"
;;
\? )
exit 1
;;
esac
done

script_path=$(dirname "${BASH_SOURCE[0]}")
if [ -z "$CONFIG" ]; then
CONFIG="${script_path}/configs"
fi

if [ -z "$TEST" ]; then
TEST="${script_path}/test_vllm.py"
fi

if [ ! -z "$GROUP" ]; then
echo "Test group is specified: $GROUP"
fi

# Parse list of configs.
for MODEL_CONFIG in "$CONFIG"/*
do
LOCAL_SUCCESS=0

echo "=== RUNNING MODEL: $MODEL_CONFIG ==="
# run test if test group is not specified or the config matching the specified test group
test_group=$(cat $MODEL_CONFIG | grep 'test_group:' | cut -d'"' -f2)
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The variable $MODEL_CONFIG is not quoted, which will cause the script to fail if a config filename contains spaces. Also, the cat command is unnecessary.

While parsing YAML with grep and cut can be fragile, a simple fix is to quote the variable and remove the cat. For a more robust solution, consider using sed or awk to handle different quoting styles or spacing in the YAML file.

Suggested change
test_group=$(cat $MODEL_CONFIG | grep 'test_group:' | cut -d'"' -f2)
test_group=$(grep 'test_group:' "$MODEL_CONFIG" | cut -d'"' -f2)

if [ -z "$GROUP" ] || [[ "$test_group" == "$GROUP" ]]; then

export TEST_DATA_FILE="$MODEL_CONFIG"
pytest \
--capture=tee-sys \
"$TEST" || LOCAL_SUCCESS=$?
echo "=== RUNNING MODEL: $MODEL_CONFIG ==="

if [[ $LOCAL_SUCCESS == 0 ]]; then
echo "=== PASSED MODEL: $MODEL_CONFIG ==="
else
echo "=== FAILED MODEL: $MODEL_CONFIG ==="
fi
export TEST_DATA_FILE="$MODEL_CONFIG"
pytest \
--capture=tee-sys \
"$TEST" || LOCAL_SUCCESS=$?

if [[ $LOCAL_SUCCESS == 0 ]]; then
echo "=== PASSED MODEL: $MODEL_CONFIG ==="
else
echo "=== FAILED MODEL: $MODEL_CONFIG ==="
fi

SUCCESS=$((SUCCESS + LOCAL_SUCCESS))
SUCCESS=$((SUCCESS + LOCAL_SUCCESS))
fi

done

Expand Down
74 changes: 44 additions & 30 deletions tests/e2e/vLLM/run_tests_in_rhaiis.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,21 @@
#!/bin/bash

usage() {
echo "Usage: $0 -c <config> -t <test> -s <save_dir>"
echo "Usage: $0 -c <config> -t <test> -g <test_group> -s <save_dir>"
exit 1
}

while getopts "c:t:s:" OPT; do
while getopts "c:t:g:s:" OPT; do
case ${OPT} in
c )
CONFIG="$OPTARG"
;;
t )
TEST="$OPTARG"
;;
g )
GROUP="$OPTARG"
;;
s )
SAVE_DIR="$OPTARG"
;;
Expand All @@ -22,20 +25,26 @@ while getopts "c:t:s:" OPT; do
esac
done

if [[ -z "$CONFIG" || -z "$TEST" || -z "$SAVE_DIR" ]]; then
echo "Error: -c, -t, and -s are required."
script_path=$(dirname "${BASH_SOURCE[0]}")
if [ -z "$CONFIG" ]; then
CONFIG="${script_path}/configs"
fi

if [ -z "$TEST" ]; then
TEST="${script_path}/test_vllm.py"
fi

if [ -z "$SAVE_DIR" ]; then
echo "Error: -s save_dir is required for rhaiis testing."
usage
fi

script_path=$(dirname "${BASH_SOURCE[0]}")
if [ -d "$CONFIG" ]; then
echo "Config is provided as a folder: $CONFIG"
CONFIGS=`ls "$CONFIG"`
elif [ -f "$CONFIG" ]; then
echo "Config is provided as a file: $CONFIG"
CONFIGS=`cat "$CONFIG"`
if [ ! -z "$GROUP" ]; then
echo "Test group is specified: $GROUP"
fi

CONFIGS=`ls "$CONFIG"`
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Using ls to populate a list of files is not recommended. It can lead to unexpected behavior with filenames that contain spaces or special characters. The subsequent loop that processes $CONFIGS is also likely to fail in such cases.

A safer and more robust way to iterate over files in a directory is to use a for loop with a glob pattern, like this:

for MODEL_CONFIG in "$CONFIG"/*
do
    # ... your logic here ...
done

This would require refactoring the existing loop structure but would make the script much more reliable.


SUCCESS=0

# Parse list of configs and add save_dir
Expand All @@ -49,30 +58,35 @@ do
save_dir=$(cat $MODEL_CONFIG | grep 'save_dir:' | cut -d' ' -f2)
model=$(cat $MODEL_CONFIG | grep 'model:' | cut -d'/' -f2)
scheme=$(cat $MODEL_CONFIG | grep 'scheme:' | cut -d' ' -f2)
test_group=$(cat $MODEL_CONFIG | grep 'test_group:' | cut -d'"' -f2)
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Similar to the other script, the variable $MODEL_CONFIG is not quoted here, and cat is unnecessary. This can cause issues if filenames have spaces. The root cause is the use of ls earlier in the script, but this line is also problematic.

Also, parsing YAML with grep and cut is fragile. Please consider quoting the variable and using a more robust parsing method.

Suggested change
test_group=$(cat $MODEL_CONFIG | grep 'test_group:' | cut -d'"' -f2)
test_group=$(grep 'test_group:' "$MODEL_CONFIG" | cut -d'"' -f2)


# add or overwrite save_dir for each model
if [[ -z "$save_dir" ]]; then
{ cat $MODEL_CONFIG; echo -e "\nsave_dir: $SAVE_DIR/$model-$scheme"; } > $CONFIG_FILE
else
{ cat $MODEL_CONFIG | grep -v 'save_dir'; echo "save_dir: $SAVE_DIR/$save_dir"; } > $CONFIG_FILE
fi
# run test if test group is not specified or the config matching the specified test group
if [ -z "$GROUP" ] || [[ "${test_group}" == "$GROUP" ]]; then

echo "=== RUNNING MODEL: $CONFIG_FILE ==="
cat $CONFIG_FILE
# add or overwrite save_dir for each model
if [[ -z "$save_dir" ]]; then
{ cat $MODEL_CONFIG; echo -e "\nsave_dir: $SAVE_DIR/$model-$scheme"; } > $CONFIG_FILE
else
{ cat $MODEL_CONFIG | grep -v 'save_dir'; echo "save_dir: $SAVE_DIR/$save_dir"; } > $CONFIG_FILE
fi

LOCAL_SUCCESS=0
export TEST_DATA_FILE="$CONFIG_FILE"
pytest \
--capture=tee-sys \
"$TEST" || LOCAL_SUCCESS=$?
echo "=== RUNNING MODEL: $CONFIG_FILE ==="
cat $CONFIG_FILE

if [[ $LOCAL_SUCCESS == 0 ]]; then
echo "=== PASSED MODEL: $CONFIG_FILE ==="
else
echo "=== FAILED MODEL: $CONFIG_FILE ==="
fi
LOCAL_SUCCESS=0
export TEST_DATA_FILE="$CONFIG_FILE"
pytest \
--capture=tee-sys \
"$TEST" || LOCAL_SUCCESS=$?

SUCCESS=$((SUCCESS + LOCAL_SUCCESS))
if [[ $LOCAL_SUCCESS == 0 ]]; then
echo "=== PASSED MODEL: $CONFIG_FILE ==="
else
echo "=== FAILED MODEL: $CONFIG_FILE ==="
fi

SUCCESS=$((SUCCESS + LOCAL_SUCCESS))
fi

done

Expand Down