[seekdb][share] Avoid idle internal table polling - #1252
Open
hnwyllmm wants to merge 401 commits into
Open
Conversation
obperf sampling shows that KVCache Wash accounts for 61.4% of process CPU during seekdb idle state, making it the biggest hotspot. Each time the wash timer (200ms) triggers, it executes: 1. `ObKVCacheStore::wash` - `refresh_score`: Iterates through all mb_handles, performing hazptr protect/release for each one. - Heap construction loop: Iterates through all mb_handles again, performing hazptr protect/release for each one. → This results in two passes, with each handle undergoing protect/release twice. 2. `ObKVCacheMap::clean_garbage_node` - Iterates over 200K buckets, acquiring a write lock (`ObBucketWLockGuard`) for each. - During idle, the vast majority of buckets are empty, so locks are acquired and immediately released, which is pure waste. → This accounts for 88% of the total wash CPU (~1280/1407 samples are in `BucketLock::wrlock/unlock`). ## Optimization 1: Merged `refresh_score` into the heap construction loop (`ob_kvcache_store.cpp`) The per-handle score decay logic from `refresh_score` was inlined into the existing heap construction loop. Each handle now performs hazptr protect/release only once, down from twice. The O(1) global calculation for `base_mb_score_` is preserved. ## Optimization 2: Fast skip for empty buckets (`ob_kvcache_map.cpp`) Before acquiring a lock, `clean_garbage_node` and `replace_fragment_node` now check if `get_bucket_node(i)` is NULL. - NULL → Skip, do not acquire the lock. - Non-NULL → Acquire the lock, then double-check before processing. Safety: Pointer reads are hardware-atomic on aarch64/x86_64; worst-case behavior reverts to the old logic. - Before optimization (PID 100137, CPU 10.3%): Wash CPU 61.4%, `clean_garbage` 58.7%. - Intermediate (PID 43062, only Optimization 1): Wash CPU 64.0%, `clean_garbage` 61.9% (Optimization 1 did not address the main bottleneck). - After optimization (PID 98980, CPU 10.7%): Wash CPU 11.5%, `clean_garbage` 7.3% (↓81%). - Functional verification: After forcing cache eviction, `clean_node_count=2~5`, and garbage nodes were correctly cleaned. ## Flame Graphs - Before optimization: http://obperf.oceanbase-dev.com/files/profile_20260517004356__work.flame.svg - After optimization: http://obperf.oceanbase-dev.com/files/profile_20260517100310__work.flame.svg
Background: obperf CPU sampling shows that in an idle seekdb instance, ObBKGDSessInActiveGuard accounts for 1.7% of samples (687/41161). This object is constructed/destructed each sleep cycle, each time involving: - thread_local diagnostic info access - reading the global variable is_ash_enabled - set_sess_inactive: rdtsc + CACHE_ALIGNED bool write - set_sess_active: rdtsc + idle time accumulation + trace_id read + ASH buffer binding A full lifecycle is triggered 100 times per second on the TimeWheel deadlock detector (10ms precision). Changes: Remove two instances of ObBKGDSessInActiveGuard. 1. ob_clock_generator.h: ObClockGenerator::usleep - Remove the inactive_guard construction before each nanosleep. - Main beneficiary: The TimeWheel stack guard in TimeWheelBase::scan goes from 303 samples to zero. 2. utility.h: ob_usleep(v, is_idle_sleep=true) - Remove the guard in the is_idle_sleep branch, merging both branches into a direct ob_usleep(v). - The is_idle_sleep parameter is kept but commented as unused for signature compatibility. - Beneficiaries: ObMultiTenant::run1, ObTimerService::run1, ObBaseLogWriter::do_flush_log, ObDDLTransController::run1, etc. Results: | Metric | Before Optimization | After Optimization | |-----------------|-------------|-------------| | BKGDSessGuard | 687 (1.7%) | 320 (0.7%) | | TimeWheel+Guard | 303 samples | 0 samples | The remaining Guard samples come from ~35 modules constructing it directly, not via the generic sleep functions. Risk Assessment: - The ASH sampler will record fewer inactive state switches during sleep periods. - No functional impact on user-visible diagnostic information. - The thread's ASH inactive flag can still be used by directly constructing the Guard. Next Steps: - The remaining 320 Guard samples come from direct construction in various modules (ob_timer_service, ob_base_log_writer, etc.) and can be optimized module-by-module as needed.
…e RTTI overhead. On the server side, run_wrapper always points to an ObTenantBase instance, making the type check performed by dynamic_cast provably redundant. Switching to static_cast removes the virtual table traversal that occurs with every timer task during tenant switching.
…Mgr Timer Task. ObPxTargetMgr executes `refresh_statistics` every 500ms. Within this, it uses the call chain `get_dummy_leader` -> `check_dummy_location_credible` -> `get_role` to determine if the current node is a Leader or Follower. Internally, `get_role` performs MTL_SWITCH + check_palf_exist + open_palf + get_role to query the Paxos role of SYS_LS. This is a CPU hotspot per tick (flame graph shows runTimerTask at 0.57%, with lib_mtl_switch accounting for 10.04% of that). SeekDB is an observer-lite single-replica mode. `ElectionImpl::get_role` always returns LEADER, and the Leader never switches. This entire Leader/Follower coordination mechanism is effectively dead code. ## Changes Made Removed the following dead code (-237 lines): 1. `get_dummy_leader` — MTL_SWITCH + nonblock_get_leader to locate the Leader. 2. `check_dummy_location_credible` — Calls `get_role` to verify cache credibility. 3. `get_role` — MTL_SWITCH + open_palf to get the Paxos role. 4. `refresh_dummy_location` — Empty function, only returns OB_SUCCESS. 5. `query_statistics` — RPC logic for Followers to report resource increments to the Leader. 6. `reset_follower_statistics` — Resets statistics on the Follower side. Removed member variables that are no longer needed: - `cluster_id_`, `dummy_cache_leader_`, `rpc_proxy_`, `need_send_refresh_all_` Simplified `refresh_statistics`: - Removed the `get_dummy_leader` call and the Follower branch. - On the first tick, executes `reset_leader_statistics` once; subsequent ticks return directly. This change applies only to single-replica (observer-lite) deployments. Multi-machine OceanBase clusters, which have a genuine need for Leader/Follower switching, cannot use this modification. - Compilation passes. - OBD deployment of a single-node instance starts normally. - MySQL connections work normally.
…n-memory atomic CAS
## Problem
The Fetcher main loop (ObCSFetcher::run1) calls try_advance_refresh_scn_ every 200ms
unconditionally, regardless of IDLE or ACTIVE mode. Under IDLE mode (no async vector
index tables), the loop iterates every 10ms, and get_refresh_scn returns GTS each time.
This results in a SQL UPDATE against __all_global_stat every 200ms even when the value
has not changed, causing unnecessary database writes.
Additionally, the Worker path (do_finish_batch_) also writes refresh_scn via SQL
within a transaction, and the virtual table query reads it via SQL — all incurring
database round-trips for a monotonically-advancing counter that only needs to be
queryable within the same process.
## Solution
Migrate refresh_scn from __all_global_stat (SQL-persisted) to in-memory management
in ObCSDispatcher using atomic variables (ATOMIC_LOAD/ATOMIC_STORE/ATOMIC_BCAS)
1. ObCSDispatcher::refresh_scn_ becomes an atomic variable with CAS-based
update_refresh_scn that advances the value only when the new value is larger.
2. Fetcher path (try_advance_refresh_scn_)
- Was: ObGlobalStatProxy::advance_change_stream_refresh_scn → SQL UPDATE
- Now: dispatcher_->update_refresh_scn → in-memory CAS, zero SQL
3. Worker path (do_finish_batch_)
- refresh_scn advancement moves from release_batch to after batch commit
- Uses dispatcher_->update_refresh_scn instead of SQL
4. Read paths
- wait_refresh_scn in ObChangeStreamMgr reads from dispatcher_->get_refresh_scn
- Virtual table ob_all_virtual_change_stream_refresh_stat reads in-memory state
- init_refresh_scn_ still loads from __all_global_stat once as recovery baseline
and allows rollback on reload (recovery semantics differ from runtime advancement)
5. Cleanup
- Removes refresh_scn_inited flag and init_refresh_scn public method
- Refactors release_batch to no longer manage refresh_scn directly
## Files Changed
- ob_change_stream_dispatcher.{h,cpp}: atomic refresh_scn_, update_refresh_scn, cleanup
- ob_change_stream_fetcher.cpp: Fetcher uses dispatcher->update_refresh_scn
- ob_change_stream_worker.cpp: Worker commits refresh_scn after batch success
- ob_change_stream_mgr.cpp: wait_refresh_scn reads from in-memory dispatcher
- ob_all_virtual_change_stream_refresh_stat.cpp: virtual table reads from memory
- wait_cs_sync.inc: mysqltest adjustment
…anual concatenation using MEMCPY. set_ext_tname is called each time a timer task executes (a hot path in handle). The original implementation used databuff_printf to format the thread name as "%s_%s", incurring runtime overhead from format string parsing and va_list variable arguments. Replaced databuff_printf with manual concatenation using STRLEN + MEMCPY: first, get the length of both strings, check that the total length does not exceed OB_EXTENED_THREAD_NAME_BUF_LEN (32), then sequentially memcpy the tname, '_', timer_name_, and '\0'. - tname (OB_THREAD_NAME_BUF_LEN=16) and timer_name_ (16) each have ≤15 valid characters. After concatenation, the total is ≤31, which always satisfies the boundary check condition of <32. - The MEMCPY / STRLEN macros are indirectly introduced via ob_define.h, adding no new dependencies. - Full compilation passes with 0 errors. - After starting a new instance, compared the log output for ext_tname; the format is completely consistent with the old implementation (e.g., TimerWK0_KVCacheWash, TimerWK0_AdvanceCKPT).
The obperf flame graph shows BlockGCTimerTask::runTimerTask consuming 3.31% of CPU, with two calls to palf_handle_impl_map_.for_each (for `get_total_used_disk_space_` and `recycle_blocks_`) each going through the ObLinkHashMap Iterator/HandleOn/revert path. In this environment, there is only 1 sys LS with a fixed palf_id of ObLSID::SYS_LS_ID (=1), making the for_each traversal entirely wasteful. Changes: 1. `get_total_used_disk_space_` — Directly call `get_palf_handle_impl(SYS_LS_ID, guard)` instead of `for_each(GetTotalUsedDiskSpace functor)`. 2. `recycle_blocks_` — Directly call `get_palf_handle_impl(SYS_LS_ID, guard)` and inline the recycling logic (base_lsn check, block GC condition check, delete_block) instead of `for_each(LogGetRecycableFileCandidate functor)`. 3. Remove the now-unused `GetTotalUsedDiskSpace` and `LogGetRecycableFileCandidate` functors (both declarations and implementations). 4. Add `#include "share/ob_ls_id.h"`.
OB_ERR_EMPTY_QUERY means the SRS table has not been fully imported (srs_cnt < 5152), so retrying is pointless — wait for the import notifier to trigger a fresh refresh instead of busy-looping every 1s.
The MemoryDump thread previously triggered a STAT_LABEL scan every 10 seconds, iterating all tenants * ctx_ids * chunks * blocks * objects to generate per-label memory statistics. This periodic full-memory walk caused significant CPU overhead on production machines. This commit sets STAT_LABEL_INTERVAL to INT64_MAX, effectively disabling periodic auto-scanning. The original code path is fully preserved. To manually trigger a memory stat scan ALTER SYSTEM REFRESH MEMORY STAT; Other on-demand dump triggers (kill -62 signal, etc/dump.config) are unaffected and continue to work normally. The STAT_LABEL_INTERVAL constant remains as a single point for future standardization — when a configurable interval parameter is introduced this constant should be replaced with that parameter.
…age/update_queue_size, and relax TIME_SLICE_PERIOD from 10ms to 1s. Flame graph analysis (obperf, 155,396 total samples) of the 10ms scheduling loop in ObMultiTenant::run1 shows two types of unnecessary periodic overhead in ObTenant::timeup. 1. update_token_usage — Every 1 second, it traverses the worker list, atomically clears idle_us_, and calculates token_usage_. However, token_usage_ and worker_us_ are only read by virtual tables (ob_all_virtual_sys_stat / ob_all_virtual_res_mgr_sys_stat) for display and do not affect any scheduling decisions. 2. update_queue_size — Every 10ms, it calls ObServerConfig::get_instance to read the tenant_task_queue_size configuration item. Under normal conditions, this configuration doesn't change, making this an ineffective polling operation. 3. TIME_SLICE_PERIOD = 10000 (10ms) — The flame graph shows ObTenant::timeup itself consumes 3.49% of total CPU, with lock operations (~47%), retry queue processing (32%), and worker recycling (19%). Being triggered every 10ms is too frequent. Relaxing this to 1s (1000000us) can significantly reduce lock contention overhead, and the semantics of tenant inspection (worker start/stop, retry replay) are not sensitive to millisecond-level delays. - Delete the update_token_usage method body and remove its call in timeup. - Delete the update_queue_size method body and remove its call in timeup. - Change get_token_usage and get_worker_time to directly return 0 (preserving interface compatibility for virtual tables). - Remove member variables: token_usage_, token_usage_check_ts_, worker_us_. - Replace update_queue_size with an inline set_queue_limit(int64_t) that directly operates on req_queue_. - Remove the corresponding members from the constructor's initialization list. - TIME_SLICE_PERIOD: 10000 → 1000000 (10ms → 1s). - Add reload_tenant_task_queue_size: acquire a read lock, read GCONF.tenant_task_queue_size, and call tenant_->set_queue_limit. - Timing interval for dumping tenant information: 10s → 1s (improves monitoring granularity). - Add a call to reload_tenant_task_queue_size at the end of ObServerReloadConfig::operator, piggybacking on the server-level configuration hot-reload path. - tenant_task_queue_size will no longer be polled every 10ms. Instead, it will be updated passively when triggered by ALTER SYSTEM SET. Configuration change path: RPC → config_mgr_->reload_config → ObServerReloadConfig::operator → ObReloadConfig::operator updates GCONF → reload_tenant_task_queue_size → set_queue_limit(GCONF.tenant_task_queue_size). - The output of update_token_usage was only used for display. After removal, virtual table queries will return 0, which does not affect system behavior. - Relaxing TIME_SLICE_PERIOD from 10ms to 1s reduces the timeup call frequency from 100Hz to 1Hz, directly cutting down the cumulative overhead from lock contention, retry drain, and worker traversal.
## Background ObBGThreadMonitor is a thread-level function execution timeout watchdog designed to monitor background thread function execution time. It was fully initialized and running (consuming ~0.52% CPU per flame graph analysis), but **zero** business code was instrumented with MonitorGuard — no threads or functions were ever registered for monitoring. Flame graph data (155,396 total samples) ObBGThreadMonitorTimerTask::runTimerTask: 805 samples (0.52%) — all self-samples, spent scanning 500 empty MonitorEntryStack arrays every second (ObClockGenerator::getClock + 2500 timestamp checks + spinlock acquire/release per cycle). ## Changes ### Deleted - src/share/ob_bg_thread_monitor.h — all classes: MonitorGuard, ObBGThreadMonitor ObBGThreadMonitorTimerTask, MonitorEntryStack, MonitorEntry BGDummyCallback, IBGCallback, MonitorCallbackWrapper ObTSIBGMonitorMemory, ObTSIBGMonitorSlotInfo, macros BG_MONITOR_GUARD(_DEFAULT) BG_NEW_CALLBACK, BG_DELETE_CALLBACK - src/share/ob_bg_thread_monitor.cpp — all implementations ### Modified - src/observer/ob_server.cpp removed #include, init/start/stop/wait/destroy lifecycle calls - src/share/ob_thread_define.h removed TG_DEF(BGThreadMonitor, BGThreadMonitor, TIMER) - src/share/CMakeLists.txt removed ob_bg_thread_monitor.cpp from build ## Risk Assessment - Zero-risk: no business code references MonitorGuard/BG_MONITOR_GUARD. Search across entire src/ confirmed no includes of ob_bg_thread_monitor.h outside the deleted files and ob_server.cpp lifecycle management. - No test files reference the framework. ## Verification - Debug build: passed (make -j80, [100%] Built target observer) - Instance deployment: deployed to ~/ob1, startup clean, SQL connectivity OK
Problem ObCSDispatcher::run1 uses dispatch_cond_.wait(100) in its idle loop. Each wakeup triggers futex kernel operations + ObWaitEventGuard + ObDiagnosticInfo::end_wait_event accounting, creating ~10 scheduling events per second even when there is no work. Flame graph (alloc view) shows this accounts for 0.95% of system-wide allocation events. Analysis The condition variable is correctly used with the standard pattern 1. Lock mutex (ObThreadCondGuard) 2. Check condition under lock 3. cond.wait(timeout) Signal is sent under the same mutex in push, so POSIX guarantees that a waiting thread is woken immediately regardless of timeout. The timeout is purely a fallback; longer timeouts do not risk losing signals. Solution 1. Increase dispatch_cond_.wait(100) to dispatch_cond_.wait(10000) idle wakeups drop from 10/sec to 0.1/sec, reducing alloc overhead by ~100x in the idle path. 2. Add dispatch_cond_.signal in stop: prevents the 10s timeout from delaying shutdown when the thread is blocked in wait. Verification - Deployed binary, confirmed idle wakeup intervals are exactly 10s (wait_duration_us = 10000067, 10000155, 10000087, 10000078). - Measured shutdown latency: 168ms (signal in stop works correctly). - Push→signal still wakes dispatcher immediately per condvar semantics (requires change stream activity to exercise directly).
…o a raw pointer. seekdb (lite deployment) always has only one LS (SYS_PALF_ID=1), which is never deleted after creation. The original ObLinkHashMap<LSKey, IPalfHandleImpl> was designed for multi-LS scenarios. Every access had to go through hash map lookup, value allocation, and reference counting, which is pure overhead in the single-LS case. Profiling revealed that LogLoopThread consumes a significant number of CPU cycles in the lambda within for_each, while in reality it's just making a simple method call on the single PalfHandleImpl. ### Data Structures (palf_env_impl.h/.cpp) - Remove the PalfHandleImplAlloc class and its 4 methods. - Remove the typedef ObLinkHashMap<LSKey, IPalfHandleImpl, PalfHandleImplAlloc>. - Remove `#include "lib/hash/ob_link_hashmap.h"`. - Replace PalfHandleImplMap + sys_ls_handle_ with `IPalfHandleImpl *single_palf_handle_`. ### Remove Reference Counting - `revert_palf_handle_impl`: No-op (handle lifetime == process lifetime). - `wait_until_reference_count_to_zero_`: Directly returns OB_SUCCESS. - `create_palf_handle_impl_`: Assigns the raw pointer. - `remove_palf_handle_impl_from_map_not_guarded_by_lock_`: Directly free + set to null. ### Simplify get_palf_handle_impl Old: hash map lookup → ref inc → check_can_be_used → revert on failure. New: null check → check_can_be_used → directly return the pointer. ### Simplify for_each (two overloads) Old: ObLinkHashMap::for_each + lambda wrapper. New: get_palf_handle_impl(SYS_PALF_ID, handle) → func → revert. ### Simplify Callers - LogLoopThread: 4 for_each calls → 4 direct method calls, removing all lambdas. - LogUpdater: for_each → directly call update_palf_stat. - `get_total_used_disk_space_`: map for_each → functor(ls_key, single_palf_handle_). - `recycle_blocks_`: map for_each → directly call `single_palf_handle_->delete_block`. - `check_can_create_palf_handle_impl_`: map.count → null check. - `check_can_update_log_disk_options_`: map.count → null check. ### Memory Safety Invariant `single_palf_handle_` is set once at startup and only set to null during shutdown. It is never freed during operation. Threads reading the pointer do not require synchronization. The `palf_meta_lock_` (RWLock) ensures writes during create/reload are visible to all subsequent reads.
…mand ## Problem ObDiskUsageReportTask::runTimerTask runs periodically to refresh an in-memory cache (result_map_) that maps (file_type, tenant_id) -> disk usage. Flame graph analysis (profile_20260520115549) shows this consumes ~1.11% CPU runTimerTask 314 samples (1.11%) count_tenant 314 self: 3 count_tenant_data 311 self: 1 get_next_tablet_ptr 310 self: 0 fetch_tablet_item 310 self: 0 for_each_value 310 self: 0 try_lock_all 310 self: 18 try_rdlock 292 (1.03%) self: 292 <-- hotspot 94% of the cost is ObLatch::try_rdlock atomic CAS operations - try_lock_all iterates ~1281 latches (10243 buckets / 8) and atomically try_rdlock's each one. This runs on every timer tick even when nobody reads the result. ## Analysis of consumers The result_map_ cache has exactly two consumers 1. get_data_disk_used_size - serves __all_virtual_unit virtual table 2. delete_tenant_usage_stat - called on tenant drop (N/A for seekdb) There is NO SQL reporting path. The comment in report_tenant_disk_usage says "update the usage table" but copies result_map_ to a local array and discards it - the reporting code was never implemented. For seekdb (single tenant, never deleted), the timer is pure overhead the only reader of result_map_ is the virtual table query. ## Solution Stop the DiskUseReport timer entirely, and make get_data_disk_used_size compute disk usage synchronously on each query. All timer-related code is deleted (not commented out) ### ob_disk_usage_reporter.h - Remove ObTimerTask inheritance (class no longer acts as a timer) - Remove runTimerTask, report_tenant_disk_usage, refresh_tenant_disk_usage count_tenant, count_tenant_slog, count_tenant_clog, count_server_slog count_server_clog, count_server_meta, count_tenant_tmp declarations - Remove ObReportResultGetter helper class - Remove const from get_data_disk_used_size ### ob_disk_usage_reporter.cpp - Delete runTimerTask, report_tenant_disk_usage, count_tenant count_tenant_slog, count_tenant_clog, count_server_slog count_server_clog, count_server_meta, count_tenant_tmp implementations - Delete commented-out set_tenant_data_usage - get_data_disk_used_size now 1. MTL_SWITCH to the target tenant 2. Calls count_tenant_data(tenant_id) to populate TENANT_DATA + TENANT_META_DATA 3. Fetches tmp file usage via ObTenantTmpFileManager 4. Reads the three needed types from result_map_ (same as before) ### ob_server.cpp - Delete TG_SCHEDULE(lib::TGDefIDs::DiskUseReport, ...) - Delete TG_START(lib::TGDefIDs::DiskUseReport) - Delete TG_STOP(lib::TGDefIDs::DiskUseReport) - Delete TG_WAIT(lib::TGDefIDs::DiskUseReport) - Delete TG_DESTROY(lib::TGDefIDs::DiskUseReport) ## Verification - Built and deployed to test instance (port 2915) - SELECT * FROM __all_virtual_unit returned data_disk_in_use = 20480000 - Observer log confirmed: no DiskUseReport messages - No errors or warnings related to ObDiskUsageReportTask in logs ## Scope & Limitations - Targeted at seekdb (single-tenant, no tenant drop). - If __all_virtual_unit is queried heavily, the on-demand computation cost will be incurred per query. This is acceptable because (a) the virtual table is typically low-QPS monitoring, and (b) single-tenant tablet count is bounded. - The result_map_ is still allocated at init and serves as the temp buffer during on-demand computation. ## Related flame graph http://obperf.oceanbase-dev.com/files/profile_20260520115549__root.flame.svg
The periodic ObCTASCleanUpTask::runTimerTask spends 100% of its CPU time in schema fetching: get_table_schema triggers full SQL-based schema fetch (fetch_table_schema via ObSchemaServiceSQLImpl) for every table in the tenant. Flame graph analysis shows this accounts for ~0.12% of total CPU samples (106/85128), all consumed by the schema fetch path. Since seekdb is single-node, two checks are unnecessary and were removed - create_host_str comparison: always matches on single node - TEMP_TAB_PROXY_RULE branch: OBProxy never exists in single-node mode is_obproxy_create_tmp_tab always returns false After removing these, all remaining schema fields used by the cleanup logic (session_id, table_type, schema_version, database_id, table_name name_case_mode, is_tmp_table, is_in_recyclebin) are available in ObSimpleTableSchemaV2. The per-table get_table_schema call is replaced with get_simple_table_schema, which reads directly from ObSchemaMgr in-memory hash table without any SQL query. This eliminates the per-table SQL schema fetch overhead entirely reducing the timer task's CPU consumption from ~106 samples to near zero.
Seekdb runs only a single sys tenant, so the Linux tc-style multi-tenant WRR scheduling tree and token-bucket rate limiter are entirely dead weight (flamegraph showed 1.31% CPU in QDiscRoot::do_thread_work). ## What changed ### Core: IO path bypasses TC queues - `ObTenantIOSchedulerV2::schedule_request` now submits IO directly to the device channel: prepare → get_device_channel → device_channel->submit - Previously: TC queue inc_ref → QSchedCallback::handle → dec_ref, with WRR scheduling and token-bucket checks in between ### Deleted: entire TC library (deps/oblib/src/lib/tc/) - 40 files, ~2900 lines: ob_tc.cpp/h, ob_tc_interface.cpp, ob_tc_limit.cpp ob_tc_stat.cpp, ob_tc_wrapper.cpp, deps/*, test/*, Makefile, README - All qdisc_*/tclimit_*/TCRequest references removed from the codebase ### Removed: ObIOManagerV2 and its lifecycle - ObIOManager::init/start/stop/destroy no longer call OB_IO_MANAGER_V2 - QSchedCallback (ITCHandler subclass) deleted ### Cleaned: ObTrafficControl and ObSharedDeviceControlV2 - Removed register_bucket, add_shared_device_limits, limit_ids_ - Removed qdisc_add_limit/qdisc_set_limit calls from add_group/inner_calc_ - Simplified add_group signature (no longer needs qid/limit_ids params) - Removed TCRequest qsched_req_ member from ObIORequest - Removed #include "lib/tc/ob_tc.h" from ob_io_define.h ### Preserved: tenant IO group tracking (independent of TC) - group_id_index_map_ / get_group_index / io_usage_ are NOT removed - They provide per-group IO statistics and have no code dependency on TC ## Verification - Compile: make -j80 in build_release, no link errors - Deploy: seekdb starts and accepts connections - Sysbench: read/write workload runs without errors - 47 files changed, +47 / -3681 lines
… mode Eliminate thread_local ObTenantBase and all tenant-switching overhead by migrating to a single global ObTenantBase pointer (g_tenant_ptr). MTL reads directly from the real ObTenant heap object — no copies, no dual objects. - ob_tenant_base.h: Replace thread_local with inline globals g_tenant_ptr and g_tenant_ctx. Both get_tenant and get_tenant_local return g_tenant_ptr — a single unified pointer. - ob_tenant_base.cpp: set_tenant and all ObTenantSwitchGuard methods gutted to no-ops. switch_to(uint64_t, bool) retains a zero-cost readiness check: g_tenant_ptr != &g_tenant_ctx, returning OB_TENANT_NOT_IN_SERVER until the real tenant is available. This allows MTL_SWITCH to safely skip its body during early init. - ob_define.h: ob_get_tenant_id returns constant OB_SYS_TENANT_ID. - ob_tenant.cpp: g_tenant_ptr = this set after create_mtl_module and before init_mtl_module. All operator= copies removed — MTL reads the real ObTenant directly via the global pointer. - ob_table_service_client.cpp: init_tenant_env gutted to no-op. - ob_dynamic_thread_pool.h: ObResetThreadTenantIdGuard gutted to no-op. - ob_all_virtual_thread.cpp: &ob_get_tenant_id → OB_SYS_TENANT_ID. g_tenant_ptr starts as &g_tenant_ctx (a dummy, no MTL services). After create_mtl_module registers all MTL services on the real ObTenant g_tenant_ptr = this. MTL_SWITCH's switch_to checks g_tenant_ptr != &g_tenant_ctx — a single pointer compare — to gate MTL access until the tenant is ready. Background threads that arrive early skip safely. - Deploy: seekdb instance starts and accepts connections - Sysbench: oltp_point_select 4-thread 30s (702K txns, 0 errors) oltp_read_write 4-thread 30s (14K txns, 2 deadlock retries) - obtest: t/stanby/basic.test passes (3/3: switchover, switchback, failover)
…k_in_set
Problem
Obperf flame graph shows find_task_in_set consumes 1680 samples (1.41%)
but the binary search comparator (CompareForSet) only compares timer_ and
task_ pointers. The 1150-sample (0.97%) TaskToken temporary object
construction — typeid(*task).name + strncpy / strnlen — is entirely
wasted work. This overhead is doubled because has_running_task calls
find_task_in_set twice (running_task_set_ + uncanceled_task_set_) for
every pop_task event.
Fix
1. Replace compare_for_set free function with transparent functor
CompareForSet (is_transparent = void, C++14 heterogeneous lookup).
Add (TaskToken*, pair) and (pair, TaskToken*) overloads so
std::lower_bound can compare directly against {timer, task} pointers.
2. In find_task_in_set, pass std::pair<const ObTimer*, const ObTimerTask*>
as the search key instead of constructing a temporary TaskToken.
3. All existing call sites (insert_unique, find) remain compatible via
the (TaskToken*, TaskToken*) overload.
Expected benefit: ~1150 samples (0.97% total CPU) eliminated from
find_task_in_set, reducing it from 1.41% to ~0.45%.
…cture ## Motivation sys_hook_impl/SYS_HOOK was originally designed to wrap blocking syscalls with WaitGuard, which set Thread::blocking_ts_ and Thread::wait_event_ so that __all_virtual_thread could display STATUS/WAIT_EVENT/LOOP_TS. This supported dynamic th_worker scaling by tracking when workers were blocked. The th_worker scaling mechanism has since been completely refactored. The hook layer, WaitGuard/JoinGuard/RpcGuard classes, and the 6 TLS variables they maintained became dead code with no remaining consumers. ## What was removed ### Thread class (thread.h/thread.cpp) - 4 Guard classes: BaseWaitGuard, WaitGuard, JoinGuard, RpcGuard - 5 WAIT_* constants: WAIT, WAIT_IN_TENANT_QUEUE, WAIT_FOR_IO_EVENT WAIT_FOR_LOCAL_RETRY, WAIT_FOR_PX_MSG - 6 TLS variables: wait_event_, blocking_ts_, loop_ts_, rpc_dest_addr_ pcode_, thread_joined_ - update_loop_ts simplified to just clear_lock ### ob_tenant_hook.cpp - Removed: sys_hook_impl, SYS_HOOK macro, in_sys_hook, ob_pthread_cond_wait ob_pthread_cond_timedwait, Linux futex_hook - Linux ob_epoll_wait: simplified to BKGDSessInActiveGuard + direct epoll_wait - ob_pthread_cond_timedwait_us: kept, Linux path now calls pthread_cond_timedwait directly ### Hook function callers - ob_pthread_cond_wait -> pthread_cond_wait (3 call sites) - ob_pthread_cond_timedwait -> pthread_cond_timedwait (2 call sites) - Removed extern declarations from ob_define.h ### futex layer - ob_futex.h: Linux now uses inline syscall(SYS_futex, ...) directly instead of futex_hook routing. Win/macOS keep futex_hook forwarding for platform emulation. - ob_futex.cpp: removed Linux weak symbol (redundant, strong symbol removed from ob_tenant_hook.cpp) - win32_unwind_stubs.c: removed ob_epoll_wait and futex_hook stubs ### ~26 WaitGuard call sites Deleted WaitGuard guard(...) constructions in 14 files without replacement ob_lock_memtable.cpp (5), ob_table_lock_service.cpp (7) ob_signal_handle.cpp, ob_tenant.cpp, ob_storage_rpc.cpp ob_local_device.cpp, obmp_query.cpp, obmp_stmt_execute.cpp obmp_stmt_prexecute.cpp, ob_data_access_service.cpp ob_lob_handler.cpp, ob_io_struct.cpp, ob_dtl_basic_channel.cpp ob_px_sqc_async_proxy.cpp, ob_rpc_proxy.ipp ### Virtual table columns - __all_virtual_thread: removed STATUS, WAIT_EVENT, LOOP_TS columns. Remaining: tid, tname, latch_wait, latch_hold, trace_id, cgroup_path, numa_node (7 cols) - Regenerated ob_inner_table_schema.*.cpp via generate_inner_table_schema.py - Removed STATUS from GV$OB_THREAD view SELECT ### ObThWorker - Removed blocking_ts accessor and blocking_ts_ member ### lua diagnostic API (ob_lua_api.cpp) - Removed loop_ts, blocking_ts, join_addr, sleep_us, rpc_dest_addr, pcode, event reads - Simplified STATUS to "Run"/"Sleep" based on sleep_us_ only - Simplified WAIT_EVENT to latch wait info only ## What was preserved - Windows/macOS ob_epoll_wait (pure platform emulation, no hook) - Windows futex_hook (WaitOnAddress/WakeByAddressSingle emulation, WaitGuard line removed) - macOS futex_hook (__ulock_wait/__ulock_wake emulation, WaitGuard line removed) - ob_pthread_cond_timedwait_us (cross-platform, with platform emulation paths) ## Verification - Build: clean compile of seekdb - Virtual table: SELECT * FROM __all_virtual_thread returns 7 columns correctly - Sysbench: oltp_point_select 30s run, ~34K TPS, 0 errors, p99 latency 0.99ms
…ond+pending bitmask ObMemoryDump used an 8-slot ObLightyQueue + pre-allocated task pool + mutex bitmask (~100 lines) to serialize multi-producer requests. Signal 62 runs in a dedicated SignalHandle thread via sigtimedwait, not a raw signal handler so all producers are ordinary thread contexts that can safely use mutex+cond. The queue/pool machinery is unnecessary. Replace queue+pool with a standard mutex+cond pattern Producer (RPC/Signal/stop) Consumer (MemoryDump thread) ──────── ──────── lock → set pending bit lock → if pending_==0: wait(10s) → signal → unlock snapshot pending_, copy task, clear unlock DUMP(b1) → handle DUMP task STAT_LABEL(b0) → handle STAT_LABEL Key decisions - pending_ is a plain int (bitmask: STAT_LABEL=1, DUMP=2), protected by mutex - DUMP has priority over STAT_LABEL to prevent debug-DUMP from being starved - cond_.wait(10s) serves dual purpose: event-driven wakeup AND timer heartbeat for periodic STAT_LABEL (every ~10s) - stop signals cond to avoid shutdown delay with large timeout - Signal 62 producer uses a stack-allocated task instead of pool allocation - deps/oblib/src/lib/alloc/memory_dump.h Removed: ObLightyQueue, task_mutex_, tasks_[8], avaliable_task_set_, TASK_NUM push, alloc_task, free_task Added: ObThreadCond cond_, int pending_, ObMemoryDumpTask pending_dump_task_ PENDING_STAT_LABEL=1, PENDING_DUMP=2, request_dump - deps/oblib/src/lib/alloc/memory_dump.cpp init: cond_.init replaces queue_.init stop: cond_.signal before TG_STOP for immediate wakeup destroy: cond_.destroy replaces queue_.destroy Deleted: push, alloc_task, free_task Added: request_dump — copy task under mutex, signal cond Simplified: generate_mod_stat_task — lock→set bit→signal→unlock Rewritten: run1 — ObThreadCondGuard RAII, snapshot+clear pending_ under lock, DUMP-first priority, timer-based STAT_LABEL heartbeat Fixed: handle no longer calls free_task (task is stack variable) - src/observer/ob_dump_task_generator.cpp Stack-allocated task replaces alloc_task/free_task/push ObMemoryDumpTask task; → fill fields → mem_dump.request_dump(task) | Path | Method | Result | |---------------|------------------------------------|--------| | RPC | ALTER SYSTEM REFRESH MEMORY STAT | Pass | | Signal 62 | kill -62 + etc/dump.config | Pass | | Timer | 10s heartbeat via cond_.wait | Pass | | Stop | kill <pid> (0.1s shutdown) | Pass | 3 files, +87/-108 lines
…PU overhead When the DDL task queue is empty (no DDL activity), the scheduler thread do_work idles with wakeup_cnt_-based cond_wait. The max idle time was 1s, causing the thread to wake every second for a no-op check even when no DDL tasks exist — consuming CPU cycles repeatedly in the flame graph. Two constants were changed ob_ddl_scheduler.h DDLIdling::get_idle_interval_us: 1s → 30s ob_ddl_task.h MAX_IDLE_TIME_US: 1s → 30s Wakeup safety (no signal loss / no processing delay) - inner_schedule_ddl_task always calls idler_.wakeup after push_task which increments wakeup_cnt_ and broadcasts the cond var. - pthread_cond_wait atomically releases mutex + sleeps, so wakeup called between check and wait is not lost (wakeup_cnt_ acts as a persistent signal, not a transient edge). - Therefore a newly-scheduled DDL task interrupts the idle immediately regardless of max idle interval. The 30s is only the upper bound when genuinely idle. Tested on seekdb single-node instance - Observed __all_ddl_task_status count 0→1→0 during CREATE INDEX confirming tasks flow through ObDDLScheduler::do_work. - Task processing completed in ~1s, well below 30s — wakeup intact.
…and hash map ## Context ObPxTargetMgr runs a 500ms periodic timer that coordinates PX worker usage across a distributed cluster: followers RPC-sync local usage to the leader, the leader aggregates into a per-server hash map. In single-server single-tenant SeekDB, all of this is wasted work. Flame graph: TimerTask::runTimerTask = 959 samples (0.81% CPU), all in DCHash traversal and map rebuild. ## What was removed (549 lines across 4 files) ### Timer & distributed coordination (306 lines) - TimerTask inner class, timer_task_ member, run_timer_task - ObPxResRefreshFunctor (always-true need_refresh_all_ dead code) - TG_START/TG_SCHEDULE/TG_STOP/TG_WAIT timer lifecycle - refresh_statistics and its entire call chain get_dummy_leader / check_dummy_location_credible / get_role / refresh_dummy_location / query_statistics (RPC, 62 lines) - rpc_proxy_, cluster_id_, dummy_cache_leader_, debug flags - PX_REFRESH_TARGET_INTERVEL_US / PX_REFRESH_CHECK_ALIVE_INTERVAL_US - Role init: FOLLOWER -> LEADER ### Data structure (243 lines) - ServerTargetUsage struct (peer/local/report, 38 lines) and its OB_SERIALIZE_MEMBER — 3-field distributed accounting protocol - hash::ObHashMap<ObAddr, ServerTargetUsage> global_target_usage_ always held exactly 1 entry (self); replaced by int64_t px_target_used_ - PX_SERVER_TARGET_BUCKET_NUM macro - get_global_target_usage -> inline get_px_target_used - ObPxGlobalResGather::operator (dead code after map removal) ### Simplified methods - apply_target: sums worker_map, single compare, SpinWLockGuard - release_target: single subtraction, no per-server map updates - get_all_target_info: hardcoded single self-entry - reset_*_statistics: px_target_used_ = 0 - update_peer_target_used: no-op stub - gather_global_target_usage: direct push, no foreach - init/reset: removed hash map create/clear ## What was preserved - apply_target / release_target — PX admission control gates concurrent queries against SET GLOBAL parallel_servers_target - reset_leader_statistics / update_peer_target_used — RPC handler stubs kept for compilation; multi-server restore points - All ObPxTargetMgr public API unchanged ## Verification (SeekDB port 12881, debug build) - Instance boots, PX queries execute normally - __all_virtual_px_target_monitor shows is_leader=1, correct target - SET GLOBAL parallel_servers_target = N syncs to monitor correctly - Zero PX timer activity in logs, zero PX errors ## How to restore multi-server PX coordination 1. Re-add TimerTask + TG_SCHEDULE in ob_px_target_mgr 2. Re-add refresh_statistics + role-detection call chain 3. Re-add rpc_proxy_ / cluster_id_ / dummy_cache_leader_ 4. Re-add ServerTargetUsage + global_target_usage_ hash map 5. Change init role back to FOLLOWER RPC handlers and ObPxTargetMgr API are ready. (cherry picked from commit ffc4f71b2386)
Problem ------- Flame graph analysis on an idle instance shows ObLockWaitMgr::run1 consumes 2.11% of total CPU, with 51.6% of that spent inside check_timeout scanning 16384 hash buckets every 100ms -- even when no transactions are waiting for locks. Root Cause ---------- FixedHash2::quick_next traverses ALL bucket headers regardless of whether they contain data. On an empty hash with LOCK_BUCKET_COUNT=16384 each check_timeout call does 16384 iterations of atomic loads and pointer checks before returning NULL. At 10 calls/second (100ms sleep) this wastes ~1% CPU on a completely idle system. The while-loop condition `!has_set_stop || !is_hash_empty` always evaluates to true during normal operation (has_set_stop is false) so is_hash_empty is never even reached to short-circuit the loop. Fix --- Two changes, both using the existing total_wait_node_ counter 1. is_hash_empty: Replace O(16384) hash scan with O(1) atomic read of total_wait_node_. This counter is already maintained correctly - INC in wait before hash_.insert - DEC in wait on wait failure, in fetch_waiter after hash_.del and in retire_node after hash_.del The INC-before-insert / DEC-after-delete ordering guarantees the counter never reads zero when nodes are present in the hash. 2. run1: Guard check_timeout with total_wait_node_ > 0 check. When no nodes are waiting, skip the entire hash scan, QSync CriticalGuard acquire/release, and fetch_deadlocked_sessions_ call. Performance Impact ------------------ Idle instance: run1 CPU drops from ~2.1% to ~0.5% (only usleep + periodic mapper dump + WaitEvent tracking remain). Instance with waiting nodes: No change. check_timeout executes normally with the same hash scan behavior as before. Safety ------ total_wait_node_ is an existing int64_t atomically maintained across all insert/delete paths. The counter covers - Normal wait → wakeup lifecycle - wait failure (has_set_stop / wakeup already happened) - Timeout / killed session / deadlock cleanup via retire_node - transform_row_lock_to_tx_lock re-insert path
Problem (from flame graph analysis) update_upper_trans_version_and_gc_sstable runs every 30s to scan all tablets and clean up old minor SSTables. The flame graph shows ~35 samples spent in check_tablet_status_for_read_all_committed — a 4-layer MDS spinlock path called per-tablet during tablet handle acquisition. Additionally, check_ls_migration_status was called inside the per-tablet loop, doing redundant LS-level revalidation for every tablet. Changes 1. Configurable tablet read mode in ObCompactionScheduleIterator - Added tablet_get_mode_ member (default: READ_ALL_COMMITED backward compatible for merge scheduling path) - Added set_tablet_get_mode setter - get_tablet_handle now uses the member instead of hardcoded READ_ALL_COMMITED 2. GC iterator uses READ_WITHOUT_CHECK mode - After build_iter in update_upper_trans_version_and_gc_sstable set mode to READ_WITHOUT_CHECK - Skips check_tablet_status_for_read_all_committed and its 4-layer MDS spinlock (MdsUnit + MdsRow), since GC only reads tablet table store metadata and does not traverse MDS transaction state 3. Hoist LS migration check out of per-tablet loop - check_ls_migration_status moved from inside the while loop to before it, so it runs once per LS instead of once per tablet - If the LS is migrating, skip all tablets in that LS Safety analysis - READ_WITHOUT_CHECK is safe for GC: the GC task only reads SSTable metadata from the tablet table store; it does not depend on MDS transaction consistency. The worst case is reading a stale table store snapshot, which means a GC opportunity is missed for one round — the next 30s iteration will catch it. - LS migration check hoisting is safe: migration status changes at LS granularity, not per-tablet. Once an LS enters migration, no tablet within that LS should be processed. Processing tablets under a migrating LS was already a no-op (the old code fell through to the empty else-if branch for ls_is_migration). Testing - Compiled and deployed to running seekdb lite instance - Verified GC task runs normally (no WARN/ERROR in observer.log) - Verified SSTable cleanup after table drop (dropped tables' SSTables were removed within one GC cycle) - Note: full major merge (merge → GC old minor SSTables) not tested on seekdb lite (compaction_scn_not_update issue); needs validation on a full OceanBase cluster Follow-up - Verify major merge + GC path on full cluster - Measure CPU reduction in flame graph after deployment
…ObVector with token-resident design ## Motivation ObSortedVector's insert_unique and remove operations trigger O(n) memmove which showed as ~1.66% CPU (wcscat/memmove) in flame graphs under load. The sorted invariant is only needed for O(1) front-peek in pop_task; an unsorted vector with full scan eliminates the memmove cost entirely. ## Design: Token-Resident Queue Instead of removing tokens from priority_task_queue_ on dispatch and re-inserting them on repeat reschedule, tokens stay in the queue for their entire lifetime. Dispatch state is encoded as scheduled_time_ = TOKEN_DISPATCHED (INT64_MAX) ### Key behavioral changes 1. schedule_task (new task) - push_back + notify (was insert_unique + conditional notify) - Always notifies because there is no sorted order to optimize against 2. schedule_task (repeat task after runTimerTask completion) - Updates scheduled_time_ in-place, token stays in queue - notify_all unconditional (was insert_unique + conditional notify on head) 3. schedule_task (non-repeat task after completion) - Scans queue to find token, swap-with-last + pop_back (O(1) removal) - Then delete_token 4. pop_task (scan loop in run1) - Full O(n) scan every tick instead of O(1) front-peek - Skips TOKEN_DISPATCHED tokens - Handles self-conflict: if running_task_token == token, skips silently - On dispatch: sets scheduled_time_ = TOKEN_DISPATCHED, token stays in queue 5. cancel_task - swap-with-last + pop_back instead of ObSortedVector::remove - Skips delete_token for TOKEN_DISPATCHED tokens (they're in running_set_) 6. stop - STEP2 skips dispatched tokens to avoid double-delete with STEP3 ### Removed - min_scheduled_time_: eliminated to avoid maintenance burden across 3 mutation paths (cancel, schedule new, schedule repeat) - compare_for_queue (free function): replaced by the iterating scan - check_clock and CLOCK_SKEW_DELTA/CLOCK_ERROR_DELTA: per-tick clock sanity check removed to reduce scheduling path overhead - ObDIActionGuard / ObBKGDSessInActiveGuard: removed from scheduling hot path - Temporary TaskToken construction in find_task_in_set CompareForSet now supports std::pair<ObTimer*, ObTimerTask*> as transparent lookup key (C++14 is_transparent) ### Trade-off At idle, pop_task's O(n) full scan is slightly more expensive than the original O(1) front-peek (~0.25% vs ~0.09% self samples). The benefit manifests under load: zero memmove on insert/remove, zero temporary allocations for set lookups, and no sorted-order maintenance.
Problem KVCache wash timer task (KVStoreWashTask) runs every ~800ms and consumes ~20% CPU when the system is idle. The hotspot is `clean_garbage_node` which traverses 200K hash bucket slots per call. Root cause `ObKVCacheStore::wash` unconditionally returns `true`, so `ObKVGlobalCache::wash` always calls `map_.clean_garbage_node` even when no memory was reclaimed. Change `ObKVCacheStore::wash` returns `reclaimed_size > 0` instead of `true`. When wash reclaims nothing (idle), `clean_garbage_node` is skipped entirely. When cache pressure exists, wash reclaims memory and `clean_garbage_node` runs normally to clean garbage hash nodes. Verification (A/B test on 8C 61GB, remote deploy + local sysbench) - obperf: clean_garbage_node 20.0% -> 0.0%, wash overhead 20.6% -> 0.9% - sysbench point_select: QPS unchanged (113K vs 114K, within noise) - Cache pressure: wash reclaims 2-35MB/cycle, memory comparable
…n cond_timedwait ObCSFetcher previously consumed ~4% idle CPU by usleep(10ms)-polling schema version in IDLE mode (no async vector index tables). Replace this with a cond_timedwait(10s) that wakes immediately on schema publish. Notification chain publish_schema → ObTenantSwitchGuard(tenant_id) → MTL_CTX → ObTenantBase::on_schema_publish [virtual] → ObTenant::on_schema_publish [override] → ObChangeStreamMgr::get_fetcher.notify_schema_changed → idle_cond_.signal Design decision: use a virtual function on ObTenantBase (overridden by ObTenant) instead of callback arrays. This keeps publish_schema unaware of business modules, ensures zero runtime overhead, and makes the call chain trivially traceable. Additional modules that need schema publish notification simply add their code to ObTenant::on_schema_publish. Double-checked locking in IDLE branch prevents TOCTOU races: re-read schema version under the cond guard before entering cond_timedwait. stop signals idle_cond_ for fast shutdown.
## Background Flame graph analysis (profile_20260520225116, obperf) on ObSimpleThreadPoolDynamicMgr::run1 shows the manager thread wakes up every 200ms to iterate all registered simple thread pools and call reap_workers. The overhead breakdown run1 total: 0.04% CPU ├─ Self (SpinRLock + loop + usleep): 59% └─ reap_workers callees: 41% ├─ ObLatchMutex::lock (workers_lock_): 21% ├─ Threads::wait: 21% ├─ ObTimeUtility::current_time: 6% ├─ usleep: 4% └─ Threads::destroy: 2% ## Change - ObSimpleThreadPoolDynamicMgr::CHECK_INTERVAL_US: 200ms -> 3s - File: deps/oblib/src/lib/thread/ob_dynamic_thread_pool.h:129 ## Rationale reap_workers only needs to clean up stopped workers. Worker lifecycle (thread exit) is on the order of seconds, not milliseconds. A 3s interval is sufficient to reclaim stopped workers without meaningful delay, while reducing the wake-up frequency by 15x (200ms -> 3s), cutting CPU overhead from ~0.04% to ~0.003%. ## Trade-off Stopped worker threads may linger up to 3s before being reaped. This is acceptable because worker stop events are rare (only during pool shrink) and the memory held by a stopped worker (thread stack + small control struct) is negligible.
…cture ## Motivation sys_hook_impl/SYS_HOOK was originally designed to wrap blocking syscalls with WaitGuard, which set Thread::blocking_ts_ and Thread::wait_event_ so that __all_virtual_thread could display STATUS/WAIT_EVENT/LOOP_TS. This supported dynamic th_worker scaling by tracking when workers were blocked. The th_worker scaling mechanism has since been completely refactored. The hook layer, WaitGuard/JoinGuard/RpcGuard classes, and the 6 TLS variables they maintained became dead code with no remaining consumers. ## What was removed ### Thread class (thread.h/thread.cpp) - 4 Guard classes: BaseWaitGuard, WaitGuard, JoinGuard, RpcGuard - 5 WAIT_* constants: WAIT, WAIT_IN_TENANT_QUEUE, WAIT_FOR_IO_EVENT WAIT_FOR_LOCAL_RETRY, WAIT_FOR_PX_MSG - 6 TLS variables: wait_event_, blocking_ts_, loop_ts_, rpc_dest_addr_ pcode_, thread_joined_ - update_loop_ts simplified to just clear_lock ### ob_tenant_hook.cpp - Removed: sys_hook_impl, SYS_HOOK macro, in_sys_hook, ob_pthread_cond_wait ob_pthread_cond_timedwait, Linux futex_hook - Linux ob_epoll_wait: simplified to BKGDSessInActiveGuard + direct epoll_wait - ob_pthread_cond_timedwait_us: kept, Linux path now calls pthread_cond_timedwait directly ### Hook function callers - ob_pthread_cond_wait -> pthread_cond_wait (3 call sites) - ob_pthread_cond_timedwait -> pthread_cond_timedwait (2 call sites) - Removed extern declarations from ob_define.h ### futex layer - ob_futex.h: Linux now uses inline syscall(SYS_futex, ...) directly instead of futex_hook routing. Win/macOS keep futex_hook forwarding for platform emulation. - ob_futex.cpp: removed Linux weak symbol (redundant, strong symbol removed from ob_tenant_hook.cpp) - win32_unwind_stubs.c: removed ob_epoll_wait and futex_hook stubs ### ~26 WaitGuard call sites Deleted WaitGuard guard(...) constructions in 14 files without replacement ob_lock_memtable.cpp (5), ob_table_lock_service.cpp (7) ob_signal_handle.cpp, ob_tenant.cpp, ob_storage_rpc.cpp ob_local_device.cpp, obmp_query.cpp, obmp_stmt_execute.cpp obmp_stmt_prexecute.cpp, ob_data_access_service.cpp ob_lob_handler.cpp, ob_io_struct.cpp, ob_dtl_basic_channel.cpp ob_px_sqc_async_proxy.cpp, ob_rpc_proxy.ipp ### Virtual table columns - __all_virtual_thread: removed STATUS, WAIT_EVENT, LOOP_TS columns. Remaining: tid, tname, latch_wait, latch_hold, trace_id, cgroup_path, numa_node (7 cols) - Regenerated ob_inner_table_schema.*.cpp via generate_inner_table_schema.py - Removed STATUS from GV$OB_THREAD view SELECT ### ObThWorker - Removed blocking_ts accessor and blocking_ts_ member ### lua diagnostic API (ob_lua_api.cpp) - Removed loop_ts, blocking_ts, join_addr, sleep_us, rpc_dest_addr, pcode, event reads - Simplified STATUS to "Run"/"Sleep" based on sleep_us_ only - Simplified WAIT_EVENT to latch wait info only ## What was preserved - Windows/macOS ob_epoll_wait (pure platform emulation, no hook) - Windows futex_hook (WaitOnAddress/WakeByAddressSingle emulation, WaitGuard line removed) - macOS futex_hook (__ulock_wait/__ulock_wake emulation, WaitGuard line removed) - ob_pthread_cond_timedwait_us (cross-platform, with platform emulation paths) ## Verification - Build: clean compile of seekdb - Virtual table: SELECT * FROM __all_virtual_thread returns 7 columns correctly - Sysbench: oltp_point_select 30s run, ~34K TPS, 0 errors, p99 latency 0.99ms
* Derive stable UUID from machine ID and base directory * fix comments * derive stable UUID from container identity --------- Co-authored-by: LINxiansheng <LINxiansheng@users.noreply.github.com>
* fix(mysqltest): wait for target major merge round * Merge remote-tracking branch 'origin/master' into issue/2026072400117720278 * Merge origin/master into issue/2026072400117720278 * Merge branch 'master' of into issue/2026072400117720278 * fix(mysqltest): stabilize macOS GIS and timing cases * test(mysqltest): restore GIS plan assertions * test(mysqltest): revert compaction wait changes --------- Co-authored-by: hnwyllmm <hnwyllmm@users.noreply.github.com> Co-authored-by: ep-12221 <ep-12221@users.noreply.github.com>
* Remove idle caching from inner SQL connection pool * Move inner SQL client tracking from connections to sessions * Move inner SQL connection lifecycle out of connection pool * Bypass connection pool for session-bound inner SQL connections * Remove inner SQL connection pools and inject lifecycle callbacks * Remove obsolete per-client inner SQL inactivity handling * Replace inner SQL lifecycle callbacks with weak symbols * Fix build regressions after inner SQL pool removal * Remove obsolete inner SQL connection and session pooling logic * Manage inner SQL connections with shared guards * Remove obsolete observer statistics and session pool remnants --------- Co-authored-by: hnwyllmm <hnwyllmm@users.noreply.github.com>
Co-authored-by: ep-12221 <ep-12221@users.noreply.github.com>
Co-authored-by: footka <footka@users.noreply.github.com>
* Reduce comparison evaluator template instantiations * Reduce datum batch hash template instantiations * Reduce fixed-double batch hash template instantiations * Reduce decimal-int cast template instantiations * Reduce object comparison operator template instantiations * Reduce decimal-int comparison template instantiations * Reduce fixed-double comparison template instantiations * Reduce decimal-int relation evaluator template instantiations * Reduce type-class comparison evaluator template instantiations * Reduce decimal-int batch scale template instantiations --------- Co-authored-by: LINxiansheng <LINxiansheng@users.noreply.github.com>
Co-authored-by: footka <footka@users.noreply.github.com>
Make DDL stats sync fail fast on stats row lock conflicts, then let the existing DDL retry path keep TAKE_EFFECT with a one-shot retry delay. Add a mysqltest reproducing concurrent gather_schema_stats and column redefinition.
Use ob_fast_localtime in ObTime2Str to avoid localtime_r/strftime timezone hot paths under verbose logging. Also fix ob_fast_localtime zero-second cache initialization and remove duplicate DTL buffer free counting. Co-authored-by: footka <footka@users.noreply.github.com>
…se#1242) Co-authored-by: LINxiansheng <LINxiansheng@users.noreply.github.com>
* remove case * Remove obsolete virtual thread diagnostic table --------- Co-authored-by: ep-12221 <ep-12221@users.noreply.github.com>
* fix: stabilize macOS mysqltest cases * test: stabilize GIS cases across platforms --------- Co-authored-by: ep-12221 <ep-12221@users.noreply.github.com>
Co-authored-by: hnwyllmm <hnwyllmm@users.noreply.github.com>
Co-authored-by: hnwyllmm <hnwyllmm@users.noreply.github.com>
Co-authored-by: ep-12221 <ep-12221@users.noreply.github.com>
Member
Author
|
The mapping Dima issue is related to inner SQL optimization. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task Description
Freeze, Timezone, DBMS Job, and DBMS Scheduler periodically query internal tables even when the tables have not changed. These idle reads continuously consume CPU and cache resources.
Solution Description
Maintain a process-local change sequence for each registered internal table. Transaction commit increments the corresponding sequence, while consumers keep their existing timers and skip internal SQL when the captured sequence is unchanged.
Seal the table registry after all modules finish initialization and before consumer timers start. Keep fail-open refresh semantics when sequence lookup fails. DBMS Job and DBMS Scheduler use full reconciliation on table changes and preserve retry/self-healing behavior for transient scheduling failures.
Passed Regressions
git diff --checkob-make seekdbob-make test_internal_table_change_notifiertest_internal_table_change_notifier: 4/4 passedUpgrade Compatibility
Compatible. The change sequence is process-local and introduces no persistent data-format or upgrade change.
Other Information
e906c59d6f0 optimize: avoid idle inner table polling.refresh_merge_inforemains periodically refreshed and is not gated by the internal-table sequence.Release Note