Skip to content

fix: init blocks when trigger is dropped and created#26

Open
strobus wants to merge 2 commits intomainfrom
fix/init-blocking
Open

fix: init blocks when trigger is dropped and created#26
strobus wants to merge 2 commits intomainfrom
fix/init-blocking

Conversation

@strobus
Copy link
Copy Markdown
Contributor

@strobus strobus commented Mar 18, 2026

Summary

  • Skip init script DDL when the outbox table already exists, avoiding ACCESS EXCLUSIVE locks that cause cascading freezes during multi-pod autoscaling
  • Add 10-second timeout to init() context as a safety net for first-time setup

Problem

NewStore() runs DROP TRIGGER + CREATE TRIGGER DDL on every pod startup. These statements require PostgreSQL ACCESS EXCLUSIVE locks on the outbox table. When a new pod starts during autoscaling:

  1. The new pod's init() requests ACCESS EXCLUSIVE on the outbox table
  2. Existing pods hold ROW EXCLUSIVE locks from active outbox dispatch transactions
  3. The ACCESS EXCLUSIVE request blocks waiting for existing transactions
  4. PostgreSQL's lock queue then blocks all subsequent operations on the table
  5. Result: cascading freeze across all pods

Fix

init() now checks information_schema.tables first. If the table exists, DDL is skipped entirely. The DDL only runs on first-time setup (when the table doesn't exist yet). A 10-second context timeout is also added so init() can never hang indefinitely.

Test plan

  • Existing tests pass (go test ./...)
  • Deploy with 1 pod, verify startup completes
  • Scale to 3+ pods simultaneously, verify no freeze
  • Monitor outbox processing continues during pod scaling

🤖 Generated with Claude Code

strobus and others added 2 commits March 18, 2026 12:02
On every pod startup, NewStore() ran init() which executes DROP TRIGGER
+ CREATE TRIGGER DDL requiring ACCESS EXCLUSIVE locks. When multiple pods
start simultaneously (autoscaling), this blocks all operations on the
outbox table across all pods, causing a cascading freeze.

Now NewStore() checks information_schema.tables first and only runs
init() DDL when the table does not yet exist. Also adds a 10-second
timeout to init() as a safety net for first-time setup.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@chatgpt-codex-connector
Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@strobus strobus added the Type: Bug Something isn't working label Mar 18, 2026
@strobus strobus requested a review from gjarmstrong March 18, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type: Bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants