Skip to content

JetStream pubsub does not recover after NATS outage; publishes keep failing with "nats: connection closed" #4563

Description

@mbasiouny

Expected Behavior

When the NATS JetStream broker becomes unavailable and then becomes healthy again, the Dapr pubsub.jetstream component should automatically recover its connection.

Subsequent publish operations should succeed without requiring a restart of the application pod or the Dapr sidecar.

Actual Behavior

When NATS becomes unavailable, publishing fails as expected.

However, after NATS becomes healthy again, all subsequent publish attempts continue to fail with:

error when publishing to topic <topic> in pubsub pubsub: nats: connection closed

The issue persists even after NATS is fully healthy again.

Restarting the application pod / Dapr sidecar immediately restores publishing.

Environment

  • Dapr runtime: 1.17.9
  • Component: pubsub.jetstream/v1
  • NATS: JetStream
  • Deployment: Kubernetes (AKS)

Steps to Reproduce the Problem

  1. Configure Dapr with the following JetStream pub/sub component:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: pubsub
spec:
  type: pubsub.jetstream
  version: v1
  metadata:
    - name: natsURL
      value: "nats://nats.nats.svc.cluster.local:4222"
    - name: token
      value: "<token>"
    - name: streamName
      value: "my-stream"
  1. Start the application and Dapr sidecar.

  2. Publish an event and verify that publishing succeeds.

  3. Stop all NATS pods.

  4. Publish another event and verify that publishing fails while NATS is unavailable.

  5. Start NATS again.

  6. Wait until all NATS pods are Running/Ready and the Kubernetes service endpoints are healthy.

Example:

nats-0   3/3   Running
nats-1   3/3   Running
nats-2   3/3   Running
  1. Without restarting the application pod or Dapr sidecar, publish another event.

  2. Observe that the publish still fails with:

nats: connection closed
  1. Restart the application pod / Dapr sidecar.

  2. Publish again and observe that publishing succeeds immediately.

Release Note

RELEASE NOTE: FIX JetStream pub/sub connection recovery after NATS broker outage.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions