Expected Behavior
When the NATS JetStream broker becomes unavailable and then becomes healthy again, the Dapr pubsub.jetstream component should automatically recover its connection.
Subsequent publish operations should succeed without requiring a restart of the application pod or the Dapr sidecar.
Actual Behavior
When NATS becomes unavailable, publishing fails as expected.
However, after NATS becomes healthy again, all subsequent publish attempts continue to fail with:
error when publishing to topic <topic> in pubsub pubsub: nats: connection closed
The issue persists even after NATS is fully healthy again.
Restarting the application pod / Dapr sidecar immediately restores publishing.
Environment
- Dapr runtime:
1.17.9
- Component:
pubsub.jetstream/v1
- NATS: JetStream
- Deployment: Kubernetes (AKS)
Steps to Reproduce the Problem
- Configure Dapr with the following JetStream pub/sub component:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.jetstream
version: v1
metadata:
- name: natsURL
value: "nats://nats.nats.svc.cluster.local:4222"
- name: token
value: "<token>"
- name: streamName
value: "my-stream"
-
Start the application and Dapr sidecar.
-
Publish an event and verify that publishing succeeds.
-
Stop all NATS pods.
-
Publish another event and verify that publishing fails while NATS is unavailable.
-
Start NATS again.
-
Wait until all NATS pods are Running/Ready and the Kubernetes service endpoints are healthy.
Example:
nats-0 3/3 Running
nats-1 3/3 Running
nats-2 3/3 Running
-
Without restarting the application pod or Dapr sidecar, publish another event.
-
Observe that the publish still fails with:
-
Restart the application pod / Dapr sidecar.
-
Publish again and observe that publishing succeeds immediately.
Release Note
RELEASE NOTE: FIX JetStream pub/sub connection recovery after NATS broker outage.
Expected Behavior
When the NATS JetStream broker becomes unavailable and then becomes healthy again, the Dapr
pubsub.jetstreamcomponent should automatically recover its connection.Subsequent publish operations should succeed without requiring a restart of the application pod or the Dapr sidecar.
Actual Behavior
When NATS becomes unavailable, publishing fails as expected.
However, after NATS becomes healthy again, all subsequent publish attempts continue to fail with:
The issue persists even after NATS is fully healthy again.
Restarting the application pod / Dapr sidecar immediately restores publishing.
Environment
1.17.9pubsub.jetstream/v1Steps to Reproduce the Problem
Start the application and Dapr sidecar.
Publish an event and verify that publishing succeeds.
Stop all NATS pods.
Publish another event and verify that publishing fails while NATS is unavailable.
Start NATS again.
Wait until all NATS pods are Running/Ready and the Kubernetes service endpoints are healthy.
Example:
Without restarting the application pod or Dapr sidecar, publish another event.
Observe that the publish still fails with:
Restart the application pod / Dapr sidecar.
Publish again and observe that publishing succeeds immediately.
Release Note
RELEASE NOTE: FIX JetStream pub/sub connection recovery after NATS broker outage.