Convex client crashes inside Celery worker context

[ortonomy]

thread '<unnamed>' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/io/driver.rs:209:27:
failed to wake I/O driver: Os { code: 9, kind: Uncategorized, message: "Bad file descriptor" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Other tasks run fine, but calling convex client results in a this crash

[nipunn1313]

Oh no, that's no good. Can you provide more information? What code are you running?

Can you run with RUST_BACKTRACE=1 environment variable to display a backtrace?

[ortonomy]

I managed to fix this by an (ugly?) workaround. I made the convex client a singleton that is only initialised when a worker task is actually called

# convex.py

from convex import ConvexClient

_client = None


def get_convex_client():
    global _client
    if _client is None:
        _client = ConvexClient(env.convex_url)
    return _client

and then used in a task

# lesson_tasks.py

@app.task(name="tasks.lesson_tasks.generate_vrps")
def generate_vrps(params: dict):
    cclient = get_convex_client()

    cclient.mutation(
        "callbacks:protectedBulkCreateLessons",
        {"apiKey": env.convex_api_key.get_secret_value()},
    )

---

I got this hint from Gemini, not sure if it's correct.

The issue stems from how the Convex client's underlying Tokio async runtime interacts with Celery's default ⁠prefork worker pool.

Root Cause: ⁠fork() and Async Runtimes

1. Celery's ⁠prefork Model: The default Celery worker pool creates worker processes by forking the main process.
2. Tokio Runtime Initialization: When you initialize ⁠ConvexClient in the global scope of your Celery module, it starts the Rust-based Tokio runtime in the main Celery process before the fork.
3. State Corruption on Fork: The Tokio runtime manages a complex set of I/O resources, including epoll/kqueue instances and network sockets, which are represented by file descriptors. When the process forks, the child worker inherits copies of these file descriptors. However, the internal state of the runtime that manages these descriptors is not safely transferred to the child. The child process's runtime is now in a corrupt state, holding file descriptors that it doesn't know how to manage correctly.
4. The Panic: When a task runs in the forked worker, the Convex client attempts to perform an I/O operation. The corrupted Tokio runtime tries to use a file descriptor that is no longer valid in its context, leading to the ⁠Bad file descriptor (EBADF) error from the OS, which triggers the Rust ⁠panic!.

This is a common "fork-safety" problem when mixing Python multiprocessing with libraries that have a complex Rust or C core managing its own event loop.

---

Not sure if help's diagnose the issue, or even if it's accurate? But yeah, I would love not to have to do the workaround. I'm comp sci education, but I've never had to mess with processes and forks and threads, so a lot of above is whoosh.

---

For completeness, the backtrace (from RUST_BACKTRACE=1)

received
thread '<unnamed>' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/io/driver.rs:209:27:
failed to wake I/O driver: Os { code: 9, kind: Uncategorized, message: "Bad file descriptor" }
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: tokio::runtime::driver::IoHandle::unpark
   4: tokio::runtime::context::with_scheduler
   5: tokio::runtime::scheduler::multi_thread::worker::<impl tokio::runtime::task::Schedule for alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::schedule
   6: tokio::runtime::task::waker::wake_by_val
   7: <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
   8: tokio::runtime::park::CachedParkThread::block_on
   9: tokio::runtime::context::runtime::enter_runtime
  10: tokio::runtime::runtime::Runtime::block_on
  11: _convex::client::PyConvexClient::__pymethod_mutation__
  12: pyo3::impl_::trampoline::trampoline
  13: _convex::client::<impl pyo3::impl_::pyclass::PyMethods<_convex::client::PyConvexClient> for pyo3::impl_::pyclass::PyClassImplCollector<_convex::client::PyConvexClient>>::py_methods::ITEMS::trampoline
  14: _method_vectorcall_VARARGS_KEYWORDS
  15: _PyObject_Vectorcall
  16: __PyEval_EvalFrameDefault
  17: __PyObject_FastCallDictTstate
  18: __PyObject_Call_Prepend
  19: _slot_tp_call
  20: __PyObject_Call
  21: __PyEval_EvalFrameDefault
  22: __PyObject_FastCallDictTstate
  23: __PyObject_Call_Prepend
  24: _slot_tp_call
  25: __PyObject_Call
  26: __PyEval_EvalFrameDefault
  27: __PyObject_FastCallDictTstate
  28: __PyObject_Call_Prepend
  29: _slot_tp_init
  30: _type_call
  31: __PyObject_MakeTpCall
  32: __PyEval_EvalFrameDefault
  33: _method_vectorcall
  34: __PyVectorcall_Call
  35: __PyEval_EvalFrameDefault
  36: __PyObject_FastCallDictTstate
  37: __PyObject_Call_Prepend
  38: _slot_tp_init
  39: _type_call
  40: __PyObject_Call
  41: __PyEval_EvalFrameDefault
  42: _method_vectorcall
  43: __PyVectorcall_Call
  44: __PyEval_EvalFrameDefault
  45: _method_vectorcall
  46: __PyVectorcall_Call
  47: __PyEval_EvalFrameDefault
  48: __PyObject_FastCallDictTstate
  49: __PyObject_Call_Prepend
  50: _slot_tp_call
  51: __PyObject_MakeTpCall
  52: __PyEval_EvalFrameDefault
  53: _PyEval_EvalCode
  54: _run_mod
  55: __PyRun_SimpleFileObject
  56: __PyRun_AnyFileObject
  57: _pymain_run_file
  58: _Py_RunMain
  59: _pymain_main
  60: _Py_BytesMain
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

aaaaand RUST_BACKTRACE=full

Task tasks.lesson_tasks.generate_vrps[a78b307e-626c-4659-90bc-295638740110] received
thread '<unnamed>' panicked at /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.36.0/src/runtime/io/driver.rs:209:27:
failed to wake I/O driver: Os { code: 9, kind: Uncategorized, message: "Bad file descriptor" }
stack backtrace:
   0:        0x108fd82e8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hcbf29052d5af526d
   1:        0x108ff35ac - core::fmt::write::hd7598e2faff23fc9
   2:        0x108fd5b6c - std::io::Write::write_fmt::hb32bb7b7a5596c85
   3:        0x108fd8140 - std::sys_common::backtrace::print::hace1623792c57522
   4:        0x108fd929c - std::panicking::default_hook::{{closure}}::h7703437986fcebac
   5:        0x108fd8f90 - std::panicking::default_hook::ha5a5cd5adea2be29
   6:        0x108fd9b54 - std::panicking::rust_panic_with_hook::hb66dc35ee3abc1e2
   7:        0x108fd9584 - std::panicking::begin_panic_handler::{{closure}}::ha4ad3e535bab9c3d
   8:        0x108fd8770 - std::sys_common::backtrace::__rust_end_short_backtrace::ha224dfe4f824568e
   9:        0x108fd92f4 - _rust_begin_unwind
  10:        0x10900b8a0 - core::panicking::panic_fmt::hfd460aec13386198
  11:        0x10900bc14 - core::result::unwrap_failed::h0c634935c0cd10d9
  12:        0x108f78e48 - tokio::runtime::driver::IoHandle::unpark::he846883c94a2ba60
  13:        0x108f7ec1c - tokio::runtime::context::with_scheduler::h20bcdd8aadabcbd5
  14:        0x108f870f8 - tokio::runtime::scheduler::multi_thread::worker::<impl tokio::runtime::task::Schedule for alloc::sync::Arc<tokio::runtime::scheduler::multi_thread::handle::Handle>>::schedule::h82e61546f2d20355
  15:        0x108f94a48 - tokio::runtime::task::waker::wake_by_val::he5b5759e7887acdf
  16:        0x108ea3ff4 - <tokio::future::poll_fn::PollFn<F> as core::future::future::Future>::poll::hb27c7fb620c0c16b
  17:        0x108e98e88 - tokio::runtime::park::CachedParkThread::block_on::h7b7fc9939d1e3461
  18:        0x108e7e1a8 - tokio::runtime::context::runtime::enter_runtime::h9efa1e3f845cf88d
  19:        0x108eab5d4 - tokio::runtime::runtime::Runtime::block_on::h4b1a7c55f7b1608e
  20:        0x108eb7334 - _convex::client::PyConvexClient::__pymethod_mutation__::hd9dcfe966605ba3d
  21:        0x108eb56b8 - pyo3::impl_::trampoline::trampoline::h5ef18ddc8d20c8f2
  22:        0x108eb5da0 - _convex::client::<impl pyo3::impl_::pyclass::PyMethods<_convex::client::PyConvexClient> for pyo3::impl_::pyclass::PyClassImplCollector<_convex::client::PyConvexClient>>::py_methods::ITEMS::trampoline::hcec58665a211d342
  23:        0x1050578c4 - _method_vectorcall_VARARGS_KEYWORDS
  24:        0x10504bde4 - _PyObject_Vectorcall
  25:        0x10515208c - __PyEval_EvalFrameDefault
  26:        0x10504b508 - __PyObject_FastCallDictTstate
  27:        0x10504c54c - __PyObject_Call_Prepend
  28:        0x1050bb8c4 - _slot_tp_call
  29:        0x10504c024 - __PyObject_Call
  30:        0x105152afc - __PyEval_EvalFrameDefault
  31:        0x10504b508 - __PyObject_FastCallDictTstate
  32:        0x10504c54c - __PyObject_Call_Prepend
  33:        0x1050bb8c4 - _slot_tp_call
  34:        0x10504c024 - __PyObject_Call
  35:        0x105152afc - __PyEval_EvalFrameDefault
  36:        0x10504b508 - __PyObject_FastCallDictTstate
  37:        0x10504c54c - __PyObject_Call_Prepend
  38:        0x1050bcb48 - _slot_tp_init
  39:        0x1050b4ee4 - _type_call
  40:        0x10504b6d0 - __PyObject_MakeTpCall
  41:        0x10515208c - __PyEval_EvalFrameDefault
  42:        0x10504e5b0 - _method_vectorcall
  43:        0x10504bcac - __PyVectorcall_Call
  44:        0x105152afc - __PyEval_EvalFrameDefault
  45:        0x10504b488 - __PyObject_FastCallDictTstate
  46:        0x10504c54c - __PyObject_Call_Prepend
  47:        0x1050bcb48 - _slot_tp_init
  48:        0x1050b4ee4 - _type_call
  49:        0x10504c024 - __PyObject_Call
  50:        0x105152afc - __PyEval_EvalFrameDefault
  51:        0x10504e5b0 - _method_vectorcall
  52:        0x10504bcac - __PyVectorcall_Call
  53:        0x105152afc - __PyEval_EvalFrameDefault
  54:        0x10504e5b0 - _method_vectorcall
  55:        0x10504bcac - __PyVectorcall_Call
  56:        0x105152afc - __PyEval_EvalFrameDefault
  57:        0x10504b488 - __PyObject_FastCallDictTstate
  58:        0x10504c54c - __PyObject_Call_Prepend
  59:        0x1050bb8c4 - _slot_tp_call
  60:        0x10504b6d0 - __PyObject_MakeTpCall
  61:        0x10515208c - __PyEval_EvalFrameDefault
  62:        0x10514d8dc - _PyEval_EvalCode
  63:        0x1051abf78 - _run_mod
  64:        0x1051aa37c - __PyRun_SimpleFileObject
  65:        0x1051a9dc0 - __PyRun_AnyFileObject
  66:        0x1051ce64c - _pymain_run_file
  67:        0x1051cdee0 - _Py_RunMain
  68:        0x1051ce130 - _pymain_main
  69:        0x1051ce1d0 - _Py_BytesMain