[Bug]: GPTQ shows poor Throughput

[namgyu-youn]

⚙️ Your current environment

The output of python collect_env.py

Operating System: `Linux-5.15.0-97-generic-x86_64-with-glibc2.35`
Python Version: `3.10.15 (main, Oct 19 2024, 16:16:22) [GCC 11.4.0]`
llm-compressor Version: `None`
compressed-tensors Version: `0.13.0`
transformers Version: `4.57.3`
torch Version: `2.9.1+cu128`
CUDA Devices: `['NVIDIA A100 80GB PCIe MIG 1g.10gb']`
AMD Devices: `None`
NPU Devices: `None

1.2B LLM Quantization shows poor thoughput/perplexity. We used AWQ for MLP layer, and GPTQ for attention block. 10 or higher throughput (requests/s) is expected, but we are getting only 5

• Checkpoint: https://huggingface.co/namgyu-youn/EXAONE-4.0-1.2B-LLMC-AWQ-W4
• benchmark result

Perplexity:
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.0098|±  |0.0044|
|     |       |strict-match    |     5|exact_match|↑  |0.0000|±  |0.0000|

Throughput: 5.01 requests/s, 5774.56 total tokens/s, 641.62 output tokens/s

🐛 Describe the bug

AWQ+GPTQ shows poor throughput (less than half of expected value).

🛠️ Steps to reproduce

tokenizer = AutoTokenizer.from_pretrained(
    "LGAI-EXAONE/EXAONE-4.0-1.2B", trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    "LGAI-EXAONE/EXAONE-4.0-1.2B", dtype=torch.bfloat16, trust_remote_code=True
)

# dataset: https://huggingface.co/datasets/LGAI-EXAONE/MANTA-1M
calibration_ds = prepare_calibration_data(tokenizer)

recipe = [
    # AWQModifier: owns MLP layers exclusively.
    AWQModifier(
        targets=[
            "re:.*gate_proj$", "re:.*up_proj$", "re:.*down_proj$",
        ],
        ignore=["lm_head"],
        scheme="W4A16",
        duo_scaling="both",
        n_grid=40,
        mappings=[
            # MLP: post-attention layernorm → gate/up
            AWQMapping(
                "re:.*post_attention_layernorm$",
                ["re:.*gate_proj$", "re:.*up_proj$"],
            ),
            # MLP: up_proj output → down_proj input
            AWQMapping(
                "re:.*up_proj$",
                ["re:.*down_proj$"],
            ),
        ],
    ),
    # GPTQ for Attention Layer
    GPTQModifier(
        targets=[
            "re:.*q_proj$", "re:.*k_proj$",
            "re:.*v_proj$", "re:.*o_proj$",
        ],
        ignore=["embed_tokens", "lm_head"],
        scheme="W4A16",
        dampening_frac=0.01,
        kv_cache_scheme={
            "num_bits": 8, "type": "float",
            "strategy": "tensor", "dynamic": False, "symmetric": True,
        },
    ),
]

oneshot(
    model=model,
    dataset=calibration_ds,
    recipe=recipe,
    max_seq_length=512,
    num_calibration_samples=512,
    shuffle_calibration_samples=False,
)

[brian-dellabetta]

Hi @namgyu-youn , just to confirm -- you are running the throughput benchmarks via vllm? vllm provides its own benchmarking tools, see thread starting here.

I wouldn't expect there to be any difference in the checkpointed model throughput compared to just running just AWQ or GPTQ or pure round-to-nearest on the same targets. If you find that to be the case, please upload two examples so we can inspect the checkpoints for discrepancies

[HDCharles]

GPTQ, Autoround and RTN (round to nearest) are quantization techniques which quantize a model or layer to a given quantization scheme, its the quantization scheme that has specific performance characteristics. All of those should have exactly the same performance as one another.

AWQ is a transform (though it leverages round to nearest as its quantization technique), it adds a rescale op that can slightly change performance characteristics.

if you're seeing different performance characteristics for different quantization techniques then something is seriously broken, can you confirm that GPTQ causes issues compared to e.g. round to nearest?

[namgyu-youn]

@brian-dellabetta @HDCharles thanks so much for fast support! I have a few question before reproducing, so it would be grateful if you can help with it, thanks in advance.

Hi @namgyu-youn , just to confirm -- you are running the throughput benchmarks via vllm? vllm provides its own benchmarking tools, see thread starting #2025 (comment).

Yes we used vLLM engine directly. I am new to llm-compressor, and not sure if I am running right recipe. Could you please confirm it? repro:

vllm bench throughput \
  --input-len 256 \
  --output-len 256 \
  --model <MODEL> \
  --num-prompts 100 \
  --max-model-len 4096 \
  --enforce-eager

And for vllm bench latency, we got reasonable result with your suggestion, thanks!

output of `vllm bench latency`

vllm bench latency --model namgyu-youn/EXAONE-4.0-1.2B-LLMC-AWQ-W4 --input-len 32 --output-len 128 --batch_size 1

Avg latency: 0.6235289263228575 seconds
10% percentile latency: 0.6223881341516971 seconds
25% percentile latency: 0.6225355863571167 seconds
50% percentile latency: 0.6229761242866516 seconds
75% percentile latency: 0.6235741060227156 seconds
90% percentile latency: 0.6239223286509514 seconds
99% percentile latency: 0.6338077741861343 seconds

---

vllm bench latency --model LGAI-EXAONE/EXAONE-4.0-1.2B --input-len 32 --output-len 128 --batch_size 1

Avg latency: 0.9997693419456481 seconds
10% percentile latency: 0.9933141581714153 seconds
25% percentile latency: 0.9936307519674301 seconds
50% percentile latency: 1.0031215250492096 seconds
75% percentile latency: 1.0045312140136957 seconds
90% percentile latency: 1.0050533927977086 seconds
99% percentile latency: 1.0080859879404307 seconds

I wouldn't expect there to be any difference in the checkpointed model throughput compared to just running just AWQ or GPTQ or pure round-to-nearest on the same targets. If you find that to be the case, please upload two examples so we can inspect the checkpoints for discrepancies

GPTQ, Autoround and RTN (round to nearest) are quantization techniques which quantize a model or layer to a given quantization scheme, its the quantization scheme that has specific performance characteristics. All of those should have exactly the same performance as one another.

AWQ is a transform (though it leverages round to nearest as its quantization technique), it adds a rescale op that can slightly change performance characteristics.

if you're seeing different performance characteristics for different quantization techniques then something is seriously broken, can you confirm that GPTQ causes issues compared to e.g. round to nearest?

Yeah AWQ is pre-scale transform before RTN, and GPTQ is a Hessian-based error reconstruction algorithm. We haven't compared all those cases, but will follow up when it's ready, thanks for your recommendation.

+1 question, is there a plan to support custom kernel like Triton? As far as I know, throughput can be enhanced in W4A16 by fused dequantization kernels (e.g., Marlin-style), where weights remain in packed W4 format during GEMM. In my initial view, llm-compressor doesn't seem to support those techniques yet (or natively supported via vLLM engine?).

[brian-dellabetta]

Hi @namgyu-youn , you can see a basic GPTQ example here. I would expect for a 1.2B parameter model, you may see a significant drop in recovery. You may want to try FP8_DYNAMIC scheme instead, as shown in this example

Yeah AWQ is pre-scale transform before RTN, and GPTQ is a Hessian-based error reconstruction algorithm. We haven't compared all those cases, but will follow up when it's ready, thanks for your recommendation.

All of this occurs during compression stage, but not during runtime. So AWQ may take longer than GPTQ to run, but the model checkpoint should run the exact same on vllm if you are using the same scheme and targeting the same layers

is there a plan to support custom kernel like Triton? As far as I know, throughput can be enhanced in W4A16 by fused dequantization kernels (e.g., Marlin-style)

Run your checkpoint on vllm. If you are using the preset schemes, it will find the best suitable kernel based on your hardware. The Marlin kernel already exists.