[feature](filecache)A 2Q-LRU mechanism for protecting hotspot data in the normal queue#57410
Open
wenzhenghu wants to merge 86 commits intoapache:masterfrom
Open
[feature](filecache)A 2Q-LRU mechanism for protecting hotspot data in the normal queue#57410wenzhenghu wants to merge 86 commits intoapache:masterfrom
wenzhenghu wants to merge 86 commits intoapache:masterfrom
Conversation
…' into normal_queue_cold_hot_seperation
ClickBench: Total hot run time: 27.96 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Contributor
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 28723 ms |
TPC-DS: Total hot run time: 183694 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 29392 ms |
TPC-DS: Total hot run time: 184487 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
TPC-H: Total hot run time: 28954 ms |
TPC-DS: Total hot run time: 184233 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Objective
In this PR, We have introduced a segmented（hot-cold） LRU mechanism To make sure that frequently accessed (“hot”) pages remain in the filecache, even as read-ahead and full table/index scans bring in new blocks that might or might not be accessed afterward.
Implementation Description
Specifically, we split the NORMAL queue of the filecache into two LRU queues, which cache hot/young data and cold/old data respectively. （The design of the hot-cold LRU mechanism references the implementation of MySQL InnoDB's midpoint LRU. For details, please refer to the link.）
The hot data remains in the original NORMAL queue. A new cold data queue, COLD_NORMAL, is added. Data that originally entered the NORMAL queue will now first enter the COLD_NORMAL queue and can only be promoted to the NORMAL queue after meeting certain conditions（reside in COLD_NORMAL queue more than normal_queue_cold_time_ms before being accessed again）.
During cache block eviction, the newly added COLD_NORMAL queue has an eviction priority between the DISPOSABLE queue and the NORMAL queue. It is important to note that cache blocks evicted from the NORMAL queue will not enter the COLD_NORMAL queue.
BE-Side Configuration Parameter
In terms of implementation, we have added three new BE configuration parameters:
enable_file_cache_normal_queue_2qlru (Boolean)
file_cache_2qlru_cold_blocks_percent (Double)
file_cache_2qlru_cold_blocks_promotion_ms (Integer)
Core Function Modifications
TPCH Test Results
The test results indicate that after enabling the cold-hot dual LRU feature, both the cache hit rate and query performance have improved, with a more significant performance improvement observed under low cache allocation ratios. The testing utilized the TPC-H benchmark. The result data is as follows:
TPCH 100 (about 25GB data should cache)
Compatibility-Related Implementation:
Related Old PR: #57353 , it is old draft pr
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)