Checklist
Describe the bug
[Bug] KV metrics report token capacity as block capacity when page_size > 1
Description
KvMetrics.kv_total_blocks and KvMetrics.kv_active_blocks are currently
reported in token units, despite their names indicating block/page units.
Current code:
kv_metrics.kv_active_blocks = int(
self.get_stats().token_usage * self.max_total_num_tokens
)
kv_metrics.kv_total_blocks = self.max_total_num_tokens
However, max_total_num_tokens is a token capacity. Elsewhere, SGLang
derives the number of KV pages as:
num_pages = self.max_total_num_tokens // self.page_size
Therefore, when page_size > 1, both kv_*_blocks values are reported in
tokens rather than blocks/pages. Their values are over-reported by
approximately page_size times. The issue is hidden when page_size == 1.
Expected behavior
kv_total_blocks should report the KV page/block count:
num_blocks = self.max_total_num_tokens // self.page_size
kv_metrics.kv_total_blocks = num_blocks
kv_active_blocks should use the same unit:
kv_metrics.kv_active_blocks = int(
self.get_stats().token_usage * num_blocks
)
Example
With:
max_total_num_tokens = 8192
page_size = 16
token_usage = 0.5
The current metrics report:
kv_total_blocks = 8192
kv_active_blocks = 4096
Expected block/page metrics should be:
kv_total_blocks = 512
kv_active_blocks = 256
Suggested test
Add a unit test with page_size > 1 (for example, 16) to verify that
kv_total_blocks == max_total_num_tokens // page_size, while retaining a
page_size == 1 case for backward compatibility.
Relevant code
Reproduction
no need to do this
Environment
no need to do this
Checklist
Describe the bug
[Bug] KV metrics report token capacity as block capacity when
page_size > 1Description
KvMetrics.kv_total_blocksandKvMetrics.kv_active_blocksare currentlyreported in token units, despite their names indicating block/page units.
Current code:
However,
max_total_num_tokensis a token capacity. Elsewhere, SGLangderives the number of KV pages as:
Therefore, when
page_size > 1, bothkv_*_blocksvalues are reported intokens rather than blocks/pages. Their values are over-reported by
approximately
page_sizetimes. The issue is hidden whenpage_size == 1.Expected behavior
kv_total_blocksshould report the KV page/block count:kv_active_blocksshould use the same unit:Example
With:
max_total_num_tokens = 8192page_size = 16token_usage = 0.5The current metrics report:
kv_total_blocks = 8192kv_active_blocks = 4096Expected block/page metrics should be:
kv_total_blocks = 512kv_active_blocks = 256Suggested test
Add a unit test with
page_size > 1(for example,16) to verify thatkv_total_blocks == max_total_num_tokens // page_size, while retaining apage_size == 1case for backward compatibility.Relevant code
Reproduction
no need to do this
Environment
no need to do this