fix: improve change stream consistency and eliminate idle statistics writes

[hnwyllmm]

Summary

This ports internal oceanbase/oceanbase-lite!1067 onto the current GitHub master.

• separate Worker-committed Change Stream progress from the in-memory refresh proof watermark
• make explicit refresh schema-consistent across schema readiness, GTS capture, LSN fencing, commit barriers, and recovery epochs
• cache the Change Stream minimum dependency LSN in memory so checkpoint can advance without periodic internal-table writes while IDLE
• move automatic optimizer statistics maintenance to the existing 15-minute timer and skip internal-table scans and writes after a successful idle reconciliation
• filter statistics-maintenance internal tables from DML and column-usage feedback

Change Stream

The persisted global state is now change_stream_applied_scn, advanced only after a Worker transaction commits. Dispatcher recovery and transaction skipping use only this committed fact.

refresh_scn_ remains an in-memory API proof watermark. Explicit refresh waits for a stable runtime schema version, Fetcher readiness, a GTS target, an LSN fence, and a dispatched-work commit barrier. Deleting the final async index is handled through the schema drained watermark, while recovery invalidates stale in-flight proofs through an epoch.

Fetcher publishes min_dep_lsn through an atomic cache. ACTIVE mode keeps the existing monotonic persistence; IDLE mode updates only memory. Checkpoint reads the cached dependency boundary directly. Recovery additionally considers PALF base LSN and rejects hints beyond the current log end.

Optimizer statistics

The built-in async gather job is marked TIMER_DRIVEN, so the generic Scheduler no longer executes it or updates its date/status fields. ObOptStatMonitorCheckTask is the single periodic trigger.

Monitoring flush, stale marking, invalid-row cleanup, and async gather use one serialized maintenance path. Epoch tracking preserves concurrent DML and enables a fast idle path after a complete reconciliation. Empty or filtered caches perform no SQL, and statistics-maintenance internal tables are excluded at both producer and flush boundaries.

Explicit procedure calls remain supported, while automatic heavy collection is dispatched off the shared timer thread.

Public port notes

• retained the public dependency-injected Change Stream log-storage and schema-publish interfaces
• added TIMER_DRIVEN where the public tree currently defines ObDBMSSchedJobInfo
• adapted Scheduler availability checks to the public SQL/module layering
• did not copy the source MR's internal-only mysqltest and mock files, which are absent from this repository

Validation

• ./build.sh release --init
• full ob-make build in build_release, without -j
• build_release/src/observer/seekdb --version
• git diff --check

The public checkout currently exposes no matching runnable CTest/mysqltest cases (ctest -N reports 0 tests). The internal-only test files from the source MR therefore could not be migrated or run here.

[Copilot]

Pull request overview

This PR ports internal changes to improve Change Stream correctness/consistency (separating committed progress from in-memory refresh proofs, adding epoch-based fencing and schema-consistent explicit refresh) and to reduce idle optimizer statistics overhead (timer-driven stats maintenance, cached monitoring epochs, and filtering of internal tables from monitoring feedback).

Changes:

• Change Stream: persist only worker-committed change_stream_applied_scn, add schema/GTS/LSN-fence/epoch-consistent explicit refresh, and cache min_dep_lsn in memory (persisting only in ACTIVE mode).
• Optimizer statistics: make async gather job timer-driven (skipped by generic Scheduler), serialize maintenance, and avoid periodic internal-table writes/scans when monitoring caches are empty or filtered.
• DBMS Scheduler: add TIMER_DRIVEN flag and executor path to run timer-driven jobs without updating scheduler state fields.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/storage/tx_storage/ob_checkpoint_service.cpp	Reads Change Stream min_dep_lsn via manager service to constrain PALF base LSN advancement.
src/sql/optimizer/stat/ob_stat_define.h	Introduces a shared 15-minute maintenance interval constant.
src/sql/optimizer/stat/ob_opt_stat_sql_service.cpp	Uses the shared maintenance interval for stat expiry calculations.
src/sql/optimizer/stat/ob_opt_stat_monitor_manager.h	Adds epoch tracking, locking, and async-gather scheduling state.
src/sql/optimizer/stat/ob_opt_stat_monitor_manager.cpp	Implements serialized maintenance, filtering, restore-on-failure, and timer-driven async gather dispatch.
src/sql/optimizer/stat/ob_dbms_stats_utils.h	Adds helpers to filter automatic monitoring/column-usage tables.
src/sql/optimizer/stat/ob_dbms_stats_utils.cpp	Implements internal-table filtering logic for monitoring/column-usage feedback.
src/sql/optimizer/stat/ob_dbms_stats_maintenance_window.h	Adds helpers for async gather job availability and timer-driven migration.
src/sql/optimizer/stat/ob_dbms_stats_maintenance_window.cpp	Marks async gather as TIMER_DRIVEN, adds migration/update logic, and restricts NEXT_DATE updates.
src/sql/optimizer/stat/ob_basic_stats_estimator.h	Adds a helper to check for remaining async-gather candidates.
src/sql/optimizer/stat/ob_basic_stats_estimator.cpp	Adds “has pending tables” check via a limited query.
src/sql/optimizer/ob_dynamic_sampling.cpp	Uses the shared maintenance interval for DS stat expiry.
src/share/ob_global_stat_proxy.h	Renames refresh SCN APIs to applied SCN APIs for Change Stream.
src/share/ob_global_stat_proxy.cpp	Implements change_stream_applied_scn persistence and lookup.
src/share/ob_debug_sync_point.h	Adds debug sync points for new Change Stream refresh/min-dep fencing paths.
src/query/api/query/scheduler/ob_scheduler_job.h	Adds TIMER_DRIVEN scheduler flag and is_timer_driven() helper.
src/pl/sys_package/ob_dbms_stats.cpp	Serializes async gather execution, flushes monitoring with epoch capture, and tracks strict completion.
src/observer/virtual_table/ob_all_virtual_change_stream_refresh_stat.cpp	Switches Change Stream refresh/min-dep reporting to in-memory manager state.
src/observer/dbms_scheduler/ob_dbms_sched_job_master.cpp	Filters timer-driven jobs out of the generic scheduler execution path.
src/observer/dbms_scheduler/ob_dbms_sched_job_executor.h	Adds timer-driven job execution API; threads through “update scheduler state” flag.
src/observer/dbms_scheduler/ob_dbms_sched_job_executor.cpp	Implements timer-driven job execution without updating scheduler state fields.
src/observer/change_stream/ob_change_stream_worker.cpp	Persists applied SCN after successful worker commit; updates in-memory refresh watermark.
src/observer/change_stream/ob_change_stream_mgr.h	Adds in-memory refresh watermark management and min-dep LSN accessor.
src/observer/change_stream/ob_change_stream_mgr.cpp	Implements schema-consistent explicit refresh with epoch/LSN fencing and in-memory refresh publication.
src/observer/change_stream/ob_change_stream_fetcher.h	Adds versioned schema-ready proof, in-memory min-dep LSN cache, and processed-end-LSN tracking.
src/observer/change_stream/ob_change_stream_fetcher.cpp	Reworks Fetcher state machine for IDLE/ACTIVE, min-dep caching/persistence rules, and schema-ready proofs.
src/observer/change_stream/ob_change_stream_dispatcher.h	Switches Dispatcher baseline to persisted applied SCN; adds recovery gate and refresh publish fence.
src/observer/change_stream/ob_change_stream_dispatcher.cpp	Implements applied-SCN init/recovery, refresh publication gating, and epoch serialization.
src/logservice/ob_log_handler.h	Adds get_base_lsn() to log handler interface.
src/logservice/ob_log_handler.cpp	Implements get_base_lsn() via PALF handle.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.

[wangyunlai-seekdb]

I found one Major failure-path issue in the PALF-entry retry path; see the inline comment.

I also re-evaluated the optimizer-statistics epoch concern and intentionally did not report it as a Major finding. Lowering ASYNC_GATHER_STALE_RATIO after an epoch has completed can defer reconsideration of already-flushed DML when no new monitored DML arrives. However, any subsequent monitored DML advances the global epoch and causes another candidate scan on the next 15-minute maintenance pass; manual gathering, direct job invocation, or restart also recovers it. The impact is limited to delayed statistics refresh for tables newly crossing the lower threshold, without data-correctness or availability impact, so it does not meet the Major threshold.

[hnwyllmm]

pipeline: 241393

[Copilot]

Copilot was unable to review this pull request because the user who requested the review is ineligible. To be eligible to request a review, you need a paid Copilot license, or your organization must enable Copilot code review.