Skip to content

fix(query): resolve spilling problems and refine memory usage in aggregation#19622

Draft
dqhl76 wants to merge 10 commits intodatabendlabs:mainfrom
dqhl76:fix-finalagg-v1
Draft

fix(query): resolve spilling problems and refine memory usage in aggregation#19622
dqhl76 wants to merge 10 commits intodatabendlabs:mainfrom
dqhl76:fix-finalagg-v1

Conversation

@dqhl76
Copy link
Collaborator

@dqhl76 dqhl76 commented Mar 26, 2026

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

fix(query): resolve spilling problems and refine memory usage in aggregation

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

dqhl76 added 10 commits March 23, 2026 13:56
Prevent BlockPartitionStream from emitting blocks larger than rows_threshold
by splitting them after build. Also adds comprehensive unit tests covering
partition, finalize_partition, and take_partitions split behavior.
…n type

Mechanical change: if let Some(block) -> for block in, matching the
updated BlockPartitionStream::finalize_partition signature.
@github-actions github-actions bot added the pr-bugfix this PR patches a bug in codebase label Mar 26, 2026
@dqhl76
Copy link
Collaborator Author

dqhl76 commented Mar 26, 2026

max_memory_limit_enabled=true
max_server_memory_usage=34359738368

SETTINGS (max_query_memory_usage = 17179869184, query_out_of_memory_behavior = 'spilling',max_aggregate_spill_level=2) select number, count(number) from numbers(2000000000) group by number ignore_result;

Before this PR, this test will OOM in final aggregation stage. The memory will continue grow.
image

After:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-bugfix this PR patches a bug in codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants