Deadlock: sync engine holds blocking parking_lot::Mutex across await; concurrent connect()/push() park every tokio worker

[leshec]

In case this issue could be useful, I thought I'd post it. I was migrating from libsql to turso sync in a DB-per-user architecture. I did some load testing, with a lot of help from LLMs, to check the DB would perform in my context before I release a beta. e.g.a classroom of 30 students submitting an exam within the same few seconds where each request opens its own connection and triggers a push for durability. I've been testing it all afternoon and seem to have found a bug, and a fix that works for me.

Here is an assisted summary...

turso = { version = "0.7.2", features = ["sync"] }, tokio multi-thread runtime.

Expected: concurrent connect()/push() calls from multiple tokio tasks queue asynchronously without blocking OS threads.
Actual: the sync engine's mutex is held across network I/O while waiters block their worker thread; once concurrency reaches the worker-thread count the whole process deadlocks permanently at 0% CPU.

Mechanism

1. The coroutines built in turso_sync_sdk_kit/src/rsapi.rs open with sync_engine.lock_arc() — a blocking parking_lot::Mutex — and hold the guard across their .await, which for a push is the whole network round-trip. Six methods do this: connect (line 424), stats (447), checkpoint (464), push_changes (481), wait_changes (497), apply_changes (519).
2. TursoDatabaseAsyncOperation::resume() drives a genawaiter::sync::Gen — a synchronous generator — so the coroutine body runs inline on whichever thread polls it (AsyncOpFuture::poll in turso/src/sync.rs calls resume()).
3. So a push takes the mutex, hits network IO, returns Pending, and releases its thread with the mutex still held. Every other engine call then parks a tokio worker thread inside lock_arc(). Once all workers are parked, nobody is left to poll the push, its IO is never consumed, and the mutex is never released.

Permanent deadlock at 0% CPU — the whole process, not just the DB calls. Triggers once concurrent engine calls ≥ worker threads, so on a 2-vCPU container (#[tokio::main] sizes to CPU count) a handful of simultaneous requests is enough.

Since Database is Clone + Send + Sync, calling it from concurrent tasks is the obvious server usage; nothing documents a serialization requirement.

Reproduction

Verified standalone (macOS arm64, 8 worker threads, remote synced DB): 4 concurrent tasks complete in <2s; 30 tasks hang forever, zero completed, 0.0% CPU. A native thread sample of the hung process shows all 8 workers in the identical stack: AsyncOpFuture::poll → resume → genawaiter → TursoDatabaseSync::connect at rsapi.rs:424 → parking_lot::RawMutex::lock_slow → parked. Sample attached.

async fn retry_busy<T, F, Fut>(mut f: F) -> T
where
    F: FnMut() -> Fut,
    Fut: std::future::Future<Output = Result<T, turso::Error>>,
{
    let mut delay = std::time::Duration::from_millis(1);
    loop {
        match f().await {
            Ok(v) => return v,
            Err(e) => {
                let text = e.to_string().to_ascii_lowercase();
                assert!(text.contains("busy") || text.contains("locked"), "non-busy error: {e}");
                tokio::time::sleep(delay).await;
                delay = (delay * 2).min(std::time::Duration::from_millis(100));
            }
        }
    }
}

#[tokio::main(flavor = "multi_thread", worker_threads = 8)]
async fn main() {
    let db = turso::sync::Builder::new_remote("repro.db")
        .with_remote_url(std::env::var("TURSO_URL").unwrap())
        .with_auth_token(std::env::var("TURSO_TOKEN").unwrap())
        .bootstrap_if_empty(true)
        .build().await.unwrap();
    let conn = db.connect().await.unwrap();
    conn.execute("CREATE TABLE IF NOT EXISTS t (id INTEGER)", ()).await.unwrap();
    drop(conn);
    db.push().await.unwrap();

    let mut set = tokio::task::JoinSet::new();
    for i in 0..30 {
        let db = db.clone();
        set.spawn(async move {
            // Each task opens its own connection, as a web server would per request.
            let conn = retry_busy(|| db.connect()).await;
            retry_busy(|| conn.execute("INSERT INTO t (id) VALUES (?)", turso::params![i])).await;
            retry_busy(|| db.push()).await;
            eprintln!("task {i} done");
        });
    }
    while let Some(r) = set.join_next().await { r.unwrap(); }
    println!("done"); // never reached once concurrency exceeds worker_threads
}

The retry_busy wrapper papers over what looks like a separate bug so the deadlock is reachable: the sync engine runs its own statements (e.g. the CDC pragma in DatabaseTape::connect) through run_stmt_* helpers that turn StepResult::Busy straight into an error, with no busy handler and no retry — so under any concurrency a bare .unwrap() panics with "database tape error: database is busy" before the deadlock can form. Happy to file that separately.

Notes

• Isolation: connects alone are fine, and pre-opened connections doing concurrent writes never contend. It's any engine call racing an in-flight push() that hangs — the mutex, not SQLite locking. Gating pushes alone is insufficient (measured: still hangs, workers park in connect).
• This is the code path the "new sync" docs name as the Rust implementation, and with_logical_mvcc_pull dispatches through the same guarded wrappers, so neither avoids it.
• Workaround we run in production: an async single-flight gate (tokio::sync::Mutex) in front of every engine call, so a waiter yields its worker instead of parking it, plus push coalescing on top.

sample-30.txt