bug: [2.0.0] Ingress controller not syncing upstream IPs on pod restart - reports success but upstreams have stale IPs

[sopacifics]

Hi team, is anyone familiar with this bug for apisix-ingress-controller 2.0.0?

• All the details here: bug: [2.0.0] Ingress controller not syncing upstream IPs on pod restart - reports success but upstreams have stale IPs apisix-ingress-controller#2699

[sopacifics]

We are having a mayor issue now, apisix-ingress-controller 1.8.x is working find with Kubernetes 1.32 but it is not working with Kubernetes 1.33

After updating Kubernetes to 1.33 apisix-ingress-controller is having issues sync endpoint when a new IP is assigned to a POD.

APISIX is still routing to the old 10.13.211.238. The controller is not syncing.

1/9/26
10:56:34.742 PM    
10.13.162.45 - - [09/Jan/2026:22:56:31 +0000] dev.cxp.cisco.com "GET /asset-tagging/api/v1/tags HTTP/1.1" 502 631 135.134 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0" 10.13.211.238:8080 502 135.105 "http://dev.cxp.cisco.com/tags"
host = ip-10-13-182-96.us-west-2.compute.internal source = /var/log/pods/apisix_apisix-596f955c4d-xxrv8_cb701631-3f1e-4b3c-a8b3-af31201d8f54/apisix/0.log sourcetype = kube:container:apisix

1/9/26
10:56:31.741 PM    
2026/01/09 22:56:31 [error] 52#52: *274336 connect() failed (110: Connection timed out) while connecting to upstream, client: 10.13.162.45, server: _, request: "GET /asset-tagging/api/v1/tags HTTP/1.1", upstream: "http://10.13.211.238:8080/tags", host: "dev.cxp.cisco.com"
host = ip-10-13-182-96.us-west-2.compute.internal source = /var/log/pods/apisix_apisix-596f955c4d-xxrv8_cb701631-3f1e-4b3c-a8b3-af31201d

kubectl get pods -n cvi-app-multiservice -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP              NODE                                          NOMINATED NODE   READINESS GATES
asset-tagging-545f459799-d8kz2       1/1     Running   0          5m17s   10.13.179.53    ip-10-13-162-45.us-west-2.compute.internal    <none>           <none>
pas-data-processor-998c84bd8-mznzg   1/1     Running   0          172m    10.13.202.183   ip-10-13-219-245.us-west-2.compute.internal   <none>           <none>

 ~ kubectl get configmap apisix-configmap -n apisix -o yaml | grep -A 2 watch_endpointslices 

      watch_endpointslices: true
      apisix_route_version: "apisix.apache.org/v2"
      enable_gateway_api: true


 ~ % kubectl logs -n apisix -l app.kubernetes.io/name=ingress-controller --tail=100 | grep -i "endpoint"

Defaulted container "ingress-controller" out of: ingress-controller, wait-apisix-admin (init)
Defaulted container "ingress-controller" out of: ingress-controller, wait-apisix-admin (init)
W0110 06:57:48.238696       1 warnings.go:70] v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice 

• The config says watch_endpointslices: true but the controller is still using v1 Endpoints (deprecated API).

This is a bug in ingress-controller 1.8.x - the watch_endpointslices setting doesn't actually work properly with Kubernetes 1.33.

Apparently, the real Fix: it is too Upgrade to 2.x (which we tried but is causing similar sync error)

Looks like that version 1.8.x is fundamentally incompatible with Kubernetes 1.33. The watch_endpointslices flag exists but doesn't properly switch to the EndpointSlice API.

Any recommendation?

[sopacifics]

@dosu anyone can help with this please?

Environment (testing in DEV):

APISIX Helm-chart v2.12.5 and AppVersion 3.14.1
apisix-ingress-controller: 2.0.0
APISIX: 3.14.1
Kubernetes 1.33

After upgrading from 1.8.4 to 2.0.0, ingress controller no longer updates APISIX upstreams when pods restart with new IPs.

No errors in apisix-ingress-controller, controller logs show "syncing all resources" with "success_count" but APISIX admin API shows stale pod IPs in upstreams.
Results in 502 errors as traffic routes to dead pods.
Expected Behavior:
When pods restart/scale and get new IPs, controller should detect endpoint changes and update corresponding upstreams in APISIX (as it did in 1.8.4).

Steps to Reproduce:

Deploy ApisixRoute with backend service
Note upstream IPs in APISIX
Delete/Restart pod to force restart with new IP
Check APISIX upstream - still has old IP
Controller logs show no errors
NO Workaround

Regression:
This worked correctly in 1.8.4 but this version doesn't seem compatible with new APISIX Helm-chart is that correct?

Apisixroute example

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: cxp-query-service:apisix.apache.org/ApisixRoute:cxp-query-service/cxp-baqs
  labels:
    app.kubernetes.io/instance: cxp-query-service
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: cxp-query-service
    app.kubernetes.io/version: 1.16.0
    helm.sh/chart: cxp-query-service-0.0.213
  name: cxp-baqs
  namespace: cxp-query-service
spec:
  http:
    - backends:
        - serviceName: cxp-baqs
          servicePort: 4000
      match:
        methods:
          - GET
        paths:
          - /cxp-baqs/api/v1/graphql
      name: graphql-playground
      plugins:
        - config:
            enable_authorization: false
          enable: true
          name: oidc-with-iam
        - config:
            regex_uri:
              - ^/cxp-baqs/api/v1/(.*)
              - /$1
          enable: true
          name: proxy-rewrite
        - config:
            count: 4
            rejected_code: 503
            time_window: 1
          enable: true
          name: limit-count
status:
  conditions:
    - message: Sync Successfully
      observedGeneration: 1
      reason: ResourcesSynced
      status: 'True'
      type: ResourcesAvailable

Route works as expected, the issue is when a new IP is assigned to the POD

Before restart

kubectl get pods

# OUTPUT:
cxp-baqs   1/1   Running   0   27m   10.13.147.219  
Route return 200

After restart

kubectl get pods

# OUTPUT:
cxp-baqs   1/1   Running   0   27m   10.13.163.151  
Error

ERRORS:

2026/01/08 23:11:37 [error] 52#52: *71860 connect() failed (113: No route to host) while connecting to upstream,
client: 10.13.204.241,
server: _,
request: "GET /cxp-baqs/api/v1 HTTP/1.1",
upstream: "http://10.13.147.219:8080/api/scopes/default", ← DEAD IP
host: "dev.cxp.cisco.com"

From APISIX access logs:

10.13.185.224 - - [08/Jan/2026:23:39:32 +0000] dev.cxp.cisco.com
"GET /ui HTTP/1.1" 502 631 9.327 "-" "Mozilla/5.0..."
10.13.147.219:8080 ← Trying this IP
502 ← All failed
3.104 ← 3 seconds per attempt

 kubectl logs -n apisix deployment/apisix-ingress-controller --tail=50

# OUTPUT:
2026-01-08T23:34:37.449Z  INFO  provider.client  syncing all resources
2026-01-08T23:34:37.449Z  INFO  provider.client  syncing resources for config  {"service_number": 16}
2026-01-08T23:34:47.376Z  INFO  provider.client  syncing all resources
2026-01-08T23:34:47.376Z  INFO  provider.client  syncing resources for config  {"service_number": 16}

Can you please help with this?