Sentry (incident): Database connection timeouts across multiple endpoints

[claude]

Database Infrastructure Incident

Incident Type: SequelizeConnectionAcquireTimeoutError cluster
Impact: Multiple endpoints affected simultaneously
Time Window: 2026-04-18 00:30:33 UTC
Total Events: 4,098+ events across multiple Sentry issues

Affected Sentry Issues

1. ETHERNAL-BACKEND-8R (Issue 243): 501 events

   • Last seen: 2026-04-18T00:30:33.839000Z
   • Link: https://sentry.tryethernal.com/organizations/sentry/issues/243/

2. ETHERNAL-BACKEND-6Z (Issue 186): 1,641 events

   • Last seen: 2026-04-18T00:30:33.827000Z
   • Link: https://sentry.tryethernal.com/organizations/sentry/issues/186/

3. ETHERNAL-BACKEND-7A (Issue 197): 1,956 events

   • Last seen: 2026-04-18T00:30:33.810000Z
   • Link: https://sentry.tryethernal.com/organizations/sentry/issues/197/

Analysis

This represents a single infrastructure incident rather than multiple application bugs. All errors:

• Share identical error class (SequelizeConnectionAcquireTimeoutError)
• Occurred within the same 30-second window
• Affect different endpoints but same root cause (database connection pool exhaustion)

Recommended Actions

1. Infrastructure Investigation:

   • Check database connection pool metrics for 2026-04-18 00:30:00 UTC
   • Review PostgreSQL logs for connection limits/timeouts
   • Check if PgBouncer experienced issues
   • Monitor for recurring patterns

2. Immediate Mitigation:

   • Verify current database connection pool health
   • Check if connection pool size needs adjustment
   • Monitor for ongoing connection acquisition issues

3. Long-term Prevention:

   • Consider connection pool size optimization
   • Add alerting for connection pool exhaustion
   • Review query patterns that might hold connections too long

Next Steps

This incident requires human investigation of the infrastructure layer rather than individual code fixes for each affected endpoint.

---

Created by Sentry Scanner - Incident Detection

[antoinedc]

Closing — no recurrence of SequelizeConnectionAcquireTimeoutError since the Apr 18 00:30 UTC spike. Pool exhaustion appears to have been a one-time event, likely mitigated by the batch of perf PRs that landed Apr 17 (#1182, #1188, #1189, #1196, #1197, #1198 — N+1 fixes in blockSync/receiptSync/processTokenTransfer and missing indexes). Will reopen or file a new incident if it recurs.