Skip to content

Improve scheduler reliability: isolate step failures, catch Redis errors in export - #153

Open
t0kubetsu wants to merge 1 commit into
D4-project:mainfrom
t0kubetsu:fix/scheduler-reliability
Open

Improve scheduler reliability: isolate step failures, catch Redis errors in export#153
t0kubetsu wants to merge 1 commit into
D4-project:mainfrom
t0kubetsu:fix/scheduler-reliability

Conversation

@t0kubetsu

Copy link
Copy Markdown
Contributor

Summary

Fixes #142. Three related reliability gaps in scheduler.py.

Changes

1. Step isolation in _run_scheduler_step

A bare step_func() call with no error handling meant one failing step silently skipped all remaining steps for the tick (report generation, job cleanup, etc.). Wrapped in try/except Exception with logger.exception — each step now fails independently.

2. Failure tracking in task_master_of_puppets

APScheduler silently swallows unhandled tick exceptions. Added an except Exception clause that logs with logger.exception and increments db.app.config["scheduler_failures"], making scheduler health observable.

3. Widen export exception handler

task_export_to_dbs only caught MeilisearchApiError and HTTPError. A Redis error during flush_batch() would propagate uncaught, causing a partial Meilisearch write with no Kvrocks update (stores diverge silently). Added redis.exceptions.RedisError, OSError, and json.JSONDecodeError to the except clause. Changed logger.error to logger.exception to preserve the traceback.

Test plan

  • Kill Kvrocks during an export tick — verify other tick steps still run and error is logged
  • Verify Meilisearch export error is logged with full traceback (not just the message)
  • Verify db.app.config["scheduler_failures"] increments on tick exception

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HIGH: Scheduler error isolation — Redis errors in flush_batch, step failures cascade, background thread silent

1 participant