DBRegistry::Shutdown() abandons other open databases to an unsafe process-exit teardown when one database's close-time flush fails (SIGABRT)

[kriszyp]

Summary

Found while rebasing PR #787 onto main and running the full pnpm test suite (60 files,
827 tests) to verify the rebase. Every in-suite test passes, but the process itself then
crashes at exit with:

rocksdb-js database registry cleanup failed: Failed to flush database during close: IO error: While open a file for appending: <tmp>/testdb-<hash>/000010.log: Permission denied
pthread lock: Invalid argument

i.e. a genuine flush IO error for one database, immediately followed by a native SIGABRT.

Confirmed native stack

gdb, ROCKSDB_ASAN=1 Release build with LD_PRELOADd ASan — ASan itself reports no heap
corruption, so this is a logic-level double-teardown rather than a buffer bug:

__GI_abort
rocksdb::port::Mutex::Lock() [clone .cold]      <- aborts, pthread_mutex_lock returns EINVAL
rocksdb::PeriodicTaskScheduler::Unregister(...)
rocksdb::DBImpl::CancelPeriodicTaskScheduler()
rocksdb::DBImpl::CancelAllBackgroundWork(bool)
rocksdb::DBImpl::CloseHelper()
rocksdb::DBImpl::Close()
rocksdb::DBImpl::WaitForCompact(WaitForCompactOptions const&)
rocksdb::StackableDB::WaitForCompact(...)
rocksdb_js::DBDeleter::operator()   src/binding/database/db_descriptor.h:54  (options.close_db = true)
std::shared_ptr<rocksdb::DB>::~shared_ptr
rocksdb_js::DBDescriptor::~DBDescriptor   src/binding/database/db_descriptor.cpp:368
...
rocksdb_js::DBRegistry::~DBRegistry   src/binding/database/db_registry.h:72   <- process-exit static destructor
__run_exit_handlers -> exit()

Root cause, as far as I traced it

DBRegistry::Shutdown() (db_registry.cpp:935) iterates every open descriptor and closes it
via closeClaimedDescriptors. Per AGENTS.md invariant 6, a close-time flush failure is
deliberately fatal for Shutdown()/PurgeAll() (failOnCompletedWithError) rather than
silently swallowed, because dropping it would hide possible data loss:

if (closeError) std::rethrow_exception(closeError);

That's by design for the one database that failed. But rethrowing exits Shutdown()
entirely — any other descriptors it had not yet reached (or was waiting on) are left in
the registry map, never gracefully closed. Binding::Init's module cleanup hook
(binding.cpp:222-233) only logs the exception (cleanup("database registry", ...)), so from
Node's perspective the env teardown "succeeds." The leftover descriptors then survive to the
process's actual C++ static destructor sweep (DBRegistry::~DBRegistry, invoked from
exit()), which force-destroys each one through DBDeleter
(db->WaitForCompact({.close_db=true})) — a path that isn't designed to safely coordinate
teardown across multiple databases the way the graceful Shutdown()/finishClose() path
is. That's where the abort happens.

What I haven't root-caused

The triggering flush error itself (Permission denied opening a WAL append file) — I
could not explain why that specific open failed. It's possible this is a genuine product bug
(some earlier operation left a descriptor/permission in a bad state) or an artifact of my dev
machine's very large, long-accumulated shared test tmp directory (8000+ leftover testdb-*
directories from unrelated sessions/days). I was not able to isolate this further within the
scope of a rebase task.

Reproduction

• Bisected with two throwaway worktrees: clean on PR Serialize database destruction with concurrent opens #787 alone (pre-rebase tip, 55 test
  files), clean on plain origin/main alone (60 test files). Only the combination (both
  histories' full test batteries, 60 files) reproduces — 3/3 on a plain build, 1/1 under
  ROCKSDB_ASAN=1 Release with LD_PRELOADd ASan.
• Always the same shape: WAL file 000010.log (db path varies, file number is consistently
  chore(deps-dev): bump @voxpelli/tsconfig from 4.2.0 to 15.0.0 #10), always inside the final DBRegistry::Shutdown() sweep at full-suite process exit.
• Have not reproduced it from a single test file or from either half of the suite alone — it
  appears to need the larger combined battery's DB volume/concurrency.

Related

• Worker-env teardown destroys transactions on the shared DBDescriptor, corrupting the heap under concurrent commits #741 (closed) — the umbrella issue for "shared DBDescriptor survives one env's teardown
  while another env is still using it," same general hazard class, different manifestation
  (worker-env teardown corrupting concurrent commits vs. this: process-exit static teardown
  after Shutdown() aborts mid-sweep). Its open children Invalidate coordinated-retry parked TSFNs when their environment exits #783, Make TransactionHandle close exclude new work and fully drain in-flight operations #784, Release pending transactions when NativeTransaction is garbage-collected #785, Audit DBHandle N-API references reachable from cross-environment shutdown #786 don't cover
  this specific path.
• commit-teardown.test.ts crashes with SIGABRT (mutex lock failed: Invalid argument) on Deno macOS CI #746 — same mutex lock ... Invalid argument / EINVAL symptom class, different subsystem
  (CommitWorker teardown on Deno/macOS CI vs. DBDeleter/PeriodicTaskScheduler here on
  Node/Linux at full-suite process exit).

Suggested next step

Reproduce in a clean tmp directory to rule out environmental noise on the triggering flush
error. If it reproduces cleanly, the actionable fix is likely in DBRegistry::Shutdown():
collect close errors from all descriptors and rethrow once at the end, rather than
abandoning the remaining descriptors to the unsafe static-destructor path on the first
failure.

— KrAIs, on behalf of @kriszyp