Skip to content

[Bug] KV metrics report token capacity as block capacity when page_size > 1 #32475

Description

@vaxilicaihouxian

Checklist

  • I searched related issues but found no solution.
  • The bug persists in the latest version.
  • Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
  • If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
  • Please use English. Otherwise, it will be closed.

Describe the bug

[Bug] KV metrics report token capacity as block capacity when page_size > 1

Description

KvMetrics.kv_total_blocks and KvMetrics.kv_active_blocks are currently
reported in token units, despite their names indicating block/page units.

Current code:

kv_metrics.kv_active_blocks = int(
    self.get_stats().token_usage * self.max_total_num_tokens
)
kv_metrics.kv_total_blocks = self.max_total_num_tokens

However, max_total_num_tokens is a token capacity. Elsewhere, SGLang
derives the number of KV pages as:

num_pages = self.max_total_num_tokens // self.page_size

Therefore, when page_size > 1, both kv_*_blocks values are reported in
tokens rather than blocks/pages. Their values are over-reported by
approximately page_size times. The issue is hidden when page_size == 1.

Expected behavior

kv_total_blocks should report the KV page/block count:

num_blocks = self.max_total_num_tokens // self.page_size
kv_metrics.kv_total_blocks = num_blocks

kv_active_blocks should use the same unit:

kv_metrics.kv_active_blocks = int(
    self.get_stats().token_usage * num_blocks
)

Example

With:

  • max_total_num_tokens = 8192
  • page_size = 16
  • token_usage = 0.5

The current metrics report:

  • kv_total_blocks = 8192
  • kv_active_blocks = 4096

Expected block/page metrics should be:

  • kv_total_blocks = 512
  • kv_active_blocks = 256

Suggested test

Add a unit test with page_size > 1 (for example, 16) to verify that
kv_total_blocks == max_total_num_tokens // page_size, while retaining a
page_size == 1 case for backward compatibility.

Relevant code

Reproduction

no need to do this

Environment

no need to do this

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions