Parallel ctx.run awaited via Promise.all fails with "awaitNext already pending" and can crash-loop (1.14.4+, incl. 1.15.0-rc.2)

[Jithin-Garapati]

Summary

Awaiting multiple ctx.run calls concurrently (native Promise.all) makes the invocation fail with:

[500 Internal] awaitNext already pending
Error: awaitNext already pending
    at ExternalProgressChannel.awaitNext (.../restate-sdk/dist/utils/external_progress_channel.js:25:69)
    at .../restate-sdk/dist/promises.js:260:41
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async .../restate-sdk/dist/promises.js:268:5
    at async .../restate-sdk/dist/promises.js:268:5
    at async PromisesExecutor.doProgress (.../restate-sdk/dist/promises.js:244:3)

Because the failure is reproduced on replay, the invocation can crash-loop indefinitely. We hit this in production: invocations stuck in backing-off with retry_count 40+, all failing on this error on every attempt.

Versions

SDK version	Result
1.14.2	no crash (but has the lost-wakeup issue that #724 fixed — we saw handlers freeze ~10 min until the server abort timeout when parallel run closures completed at the wrong moment)
1.14.4	crashes
1.14.5	crashes
1.15.0-rc.2	crashes

Server: restate 1.6.x. Node v26. Also reproduced against restate-cli 1.6.2 local dev server on macOS.

Reproduction

import * as restate from "@restatedev/restate-sdk";

const svc = restate.object({
  name: "MinimalParallelRun",
  handlers: {
    run: async (ctx) => {
      for (let i = 0; i < 5; i++) {
        // sequential step first (like a durable model call)...
        await ctx.run(`model-${i}`, async () => "model");
        // ...then parallel steps awaited with native Promise.all (like AI SDK
        // executing multiple tool calls concurrently)
        await Promise.all([
          ctx.run(`a-${i}`, async () => "a"),
          ctx.run(`b-${i}`, async () => "b"),
          ctx.run(`c-${i}`, async () => "c"),
        ]);
      }
      return "done";
    },
  },
});

restate.endpoint().bind(svc).listen(9082);

restate dev   # or any restate server
node service.mjs
curl -X POST http://127.0.0.1:9070/deployments -H 'Content-Type: application/json' -d '{"uri":"http://127.0.0.1:9082"}'
curl -X POST http://127.0.0.1:8080/MinimalParallelRun/test/run -H 'Content-Type: application/json' -d '{}'

At this small scale the invocation eventually completes after several crash/replay cycles (~9s instead of ~50ms; the endpoint log shows repeated awaitNext already pending + Replaying invocation). With larger journals and wider parallelism (our production agent: hundreds of journal entries, 2–8 parallel tool ctx.runs per step) the lucky interleaving never happens and the invocation crash-loops until retries are exhausted.

Suspected cause

ExternalProgressChannel.awaitNext() (introduced in #724) rejects when a second consumer awaits concurrently. When user code awaits several RestatePromises at once, multiple PromisesExecutor.doProgress loops run concurrently and both reach await this.externalProgressChannel.awaitNext() (promises.js:260) → the second one throws.

The stack trace shows two concurrent doProgressInner frames (the doubled promises.js:268 async frames).

Impact

Any handler that awaits more than one ctx.run concurrently. Notably this includes @restatedev/vercel-ai-middleware users: the AI SDK executes tool calls in parallel when a model emits multiple tool calls in one step, so any agent using durableCalls + parallel tools is affected on >= 1.14.4.

This currently leaves no released version that handles parallel ctx.run correctly: 1.14.2 loses wakeups (long stalls until abort timeout), 1.14.4+ crash-loops.

[marcus-sa]

You have to use RestatePromise.all instead of Promise.all.