Connection timeout (2013) during authentication after ~1 hour of bigtrans_lite stability test, preceded by numerous minor compaction scheduling failures (-4677)

[cms-cms]

Environment

• Version: observer (OceanBase seekdb)
• Revision: 1-f84169fbd80d020887d038cdb0892703e28833f3
• Build Branch: HEAD
• Build Time: Aug 10 2026 10:45:34
• Build Flags: RelWithDebInfo
• Is this an RD-provided temporary version?: No
• Reproducibility: To be determined (必现/偶现/不确定/其他)
• Test Changes: To be determined (增加了xx测试场景/换了新机器/新跑了xx工具/替换新版本后才出的)
• Other environmental anomalies/changes: To be determined (否/clog盘满过/手工做过。。/注入过xx异常)

Description

During a long-running bigtrans_lite workload, after approximately one hour, a connection timeout (MySQL error 2013) occurred during the authentication phase. The failure was preceded by a large number of minor compaction scheduling attempts returning error code -4677.

Steps to Reproduce / Scenario

Please provide detailed steps, workload description, actions taken before the issue, and any test changes.
Strongly discouraged: Only posting ERROR logs or stack traces without context.

Environment Information

• Problem Tenant Connection String: To be provided.
• Sys Tenant Connection String: To be provided.
• Deployment Info: To be provided.
• Is the scene preserved?: Yes, the database instance is still running.

  host: [INTERNAL_IP]
  port: 2830
  observer pid: 1382607
  

• Error Observed:

  ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 110
  

Logs & Evidence

Database Log Evidence Before Failure

Seconds before the connection timeout, the observer logs showed a large number of entries:

schedule_tablet_minor (ob_tablet_scheduler.cpp:1004)
try_schedule_compaction_after_mini (ob_tablet_merge_ctx.cpp:215)

failed to schedule tablet merge
tmp_ret=-4677

failed to schedule special tablet minor merge
errcode=-4677

Log scan statistics from the failure window:

lines=22907398
alarm=1007
alarm_scan_capped=1
truncated=7

This indicates an extremely high volume of logs and numerous compaction scheduling failures during the failure window. While the timing is highly correlated, existing logs alone cannot definitively assert that -4677 is the direct root cause of error 2013. Kernel development needs to confirm the exact meaning of -4677 and check for potential DAG, scheduling queue, or resource exhaustion issues.

Database Connection

Connection command used:

obclient -h[INTERNAL_IP] -P 2830 -uroot@sys -A -Dtest

The database instance is still preserved:

host: [INTERNAL_IP]
port: 2830
observer pid: 1382607

Workload and Log Locations

Workload logs:

/data/2/mk_resource/seekdb-gitlab/bigtrans_lite/runs/pipeline-189961-job-919129/scenario/singlejob/bigtrans_lite/obtrxtest/obtest.log
/data/2/mk_resource/seekdb-gitlab/bigtrans_lite/runs/pipeline-189961-job-919129/scenario/singlejob/bigtrans_lite/bigtrans_lite.result
/data/2/mk_resource/seekdb-gitlab/bigtrans_lite/runs/pipeline-189961-job-919129/scenario/singlejob/bigtrans_lite/conf/ds.properties

Database log directory:

/data/1/seekdb-gitlab/bigtrans_lite-log/pipeline-189961-job-919129

Due to rapid log rotation, the initial failure logs may have been recycled online. GitLab artifacts have preserved a failure snapshot:

physical_artifacts/bigtrans_lite/linux/healthcheck.last.log
physical_artifacts/bigtrans_lite/linux/watchdog.log
physical_artifacts/bigtrans_lite/linux/live-failure-diagnostics.log
physical_artifacts/bigtrans_lite/linux/database-log-alarm/report.txt
physical_artifacts/bigtrans_lite/linux/database-log-alarm/report.json
physical_artifacts/bigtrans_lite/linux/database-log/

Job Link: [bigtrans_lite Job 919129](bigtrans_lite Job 919129)

Suggested Focus for Development

• The error code -4677 and its triggering conditions.
• Whether minor compaction DAG/scheduling resources were exhausted.
• Why the observer process was alive but authentication requests could not complete before the timeout.
• CPU, I/O, thread pool, DAG queue, and memory state during the failure window.
• Whether there was a log storm and cascading blockage caused by compaction scheduling failures.

Other

For issues reported by external business teams, please correctly select the "Work Item Type" based on the actual situation.