Claude/kubernetes migration plan kq jw d (#358)

t0mdavid-m · claude · web-flow · commit 93170fa84563 · 2026-04-04T14:11:11.000+02:00
* Add Kubernetes manifests and CI workflows for de.NBI migration Decompose the monolithic Docker container into Kubernetes workloads: - Streamlit Deployment with health probes and session affinity - Redis Deployment + Service for job queue - RQ Worker Deployment for background workflows - CronJob for workspace cleanup - Ingress with WebSocket support and cookie-based sticky sessions - Shared PVC (ReadWriteMany) for workspace data - ConfigMap for runtime configuration (replaces build-time settings) - Kustomize base + template-app overlay for multi-app deployment Code changes: - Remove unsafe enableCORS=false and enableXsrfProtection=false from config.toml - Make workspace path configurable via WORKSPACES_DIR env var in clean-up-workspaces.py CI/CD: - Add build-and-push-image.yml to push Docker images to ghcr.io - Add k8s-manifests-ci.yml for manifest validation and kind integration tests https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix kubeconform validation to skip kustomization.yaml kustomization.yaml is a Kustomize config file, not a standard K8s resource, so kubeconform has no schema for it. Exclude it via -ignore-filename-pattern. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Add matrix strategy to test both Dockerfiles in integration tests The integration-test job now uses a matrix with Dockerfile_simple and Dockerfile. Each matrix entry checks if its Dockerfile exists before running — all steps are guarded with an `if` condition so they skip gracefully when a Dockerfile is absent. This allows downstream forks that only have one Dockerfile to pass CI without errors. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Adapt K8s base manifests for de.NBI Cinder CSI storage - Switch workspace PVC from ReadWriteMany to ReadWriteOnce with cinder-csi storage class (required by de.NBI KKP cluster) - Increase PVC storage to 500Gi - Add namespace: openms to kustomization.yaml - Reduce pod resource requests (1Gi/500m) and limits (8Gi/4 CPU) so all workspace-mounting pods fit on a single node https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Add pod affinity rules to co-locate all workspace pods on same node The workspaces PVC uses ReadWriteOnce (Cinder CSI block storage) which requires all pods mounting it to run on the same node. Without explicit affinity rules, the scheduler was failing silently, leaving pods in Pending state with no events. Adds a `volume-group: workspaces` label and podAffinity with requiredDuringSchedulingIgnoredDuringExecution to streamlit deployment, rq-worker deployment, and cleanup cronjob. This ensures the scheduler explicitly co-locates all workspace-consuming pods on the same node. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix CI: wait for ingress-nginx admission webhook before deploying The controller pod being Ready doesn't guarantee the admission webhook service is accepting connections. Add a polling loop that waits for the webhook endpoint to have an IP assigned before applying the Ingress resource, preventing "connection refused" errors during kustomize apply. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix CI: add -n openms namespace to integration test steps The kustomize overlay deploys into the openms namespace, but the verification steps (Redis wait, Redis ping, deployment checks) were querying the default namespace, causing "no matching resources found". https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix CI: retry kustomize deploy for webhook readiness Replace the unreliable endpoint-IP polling with a retry loop on kubectl apply (up to 5 attempts with backoff). This handles the race where the ingress-nginx admission webhook has an endpoint IP but isn't yet accepting TCP connections. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix REDIS_URL to use prefixed service name in overlay Kustomize namePrefix renames the Redis service to template-app-redis, but the REDIS_URL env var in streamlit and rq-worker deployments still referenced the unprefixed name "redis", causing the rq-worker to CrashLoopBackOff with "Name or service not known". Add JSON patches in the overlay to set the correct prefixed hostname. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Add Traefik IngressRoute for direct LB IP access The cluster uses Traefik, not nginx, so the nginx Ingress annotations are ignored. Add a Traefik IngressRoute with PathPrefix(/) catch-all routing and sticky session cookie for Streamlit session affinity. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix CI: skip Traefik IngressRoute CRD in validation and integration tests kubeconform doesn't know the Traefik IngressRoute CRD schema, and the kind cluster in integration tests doesn't have Traefik installed. Skip the IngressRoute in kubeconform validation and filter it out with yq before applying to the kind cluster. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * Fix IngressRoute service name for kustomize namePrefix Kustomize namePrefix doesn't rewrite service references inside CRDs, so the IngressRoute was pointing to 'streamlit' instead of 'template-app-streamlit', causing Traefik to return 404. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * fix: use ConfigMap as settings override instead of full replacement The ConfigMap was replacing the entire settings.json, losing keys like "version" and "repository-name" that the app expects (causing KeyError). Now the ConfigMap only contains deployment-specific overrides, which are merged into the Docker image's base settings.json at container startup using jq. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ * fix: add set -euo pipefail to fail fast on settings merge error Addresses CodeRabbit review: if jq merge fails, the container should not start with unmerged settings. https://claude.ai/code/session_01RNJ3dVjV1VTHcC9ugE3FQJ --------- Co-authored-by: Claude <noreply@anthropic.com>
diff --git a/.github/workflows/k8s-manifests-ci.yml b/.github/workflows/k8s-manifests-ci.yml
@@ -21,7 +21,10 @@ jobs:
 
       - name: Validate K8s manifests (base)
         run: |
-          kubeconform -summary -strict -kubernetes-version 1.28.0 -ignore-filename-pattern 'kustomization.yaml' k8s/base/*.yaml
+          kubeconform -summary -strict -kubernetes-version 1.28.0 \
+            -ignore-filename-pattern 'kustomization.yaml' \
+            -ignore-filename-pattern 'traefik-ingressroute.yaml' \
+            k8s/base/*.yaml
 
       - name: Install kubectl
         uses: azure/setup-kubectl@v3
@@ -33,7 +36,7 @@ jobs:
 
       - name: Validate kustomized output
         run: |
-          kubectl kustomize k8s/overlays/template-app/ | kubeconform -summary -strict -kubernetes-version 1.28.0
+          kubectl kustomize k8s/overlays/template-app/ | kubeconform -summary -strict -kubernetes-version 1.28.0 -skip IngressRoute
 
   integration-test:
     runs-on: ubuntu-latest
@@ -83,7 +86,9 @@ jobs:
       - name: Deploy with Kustomize
         if: steps.check.outputs.exists == 'true'
         run: |
+          # Filter out Traefik CRDs (kind cluster uses nginx, not Traefik)
           kubectl kustomize k8s/overlays/template-app/ | \
+            yq 'select(.kind != "IngressRoute")' | \
             sed 's|imagePullPolicy: IfNotPresent|imagePullPolicy: Never|g' > /tmp/manifests.yaml
           for i in 1 2 3 4 5; do
             if kubectl apply -f /tmp/manifests.yaml; then
diff --git a/k8s/base/configmap.yaml b/k8s/base/configmap.yaml
@@ -3,37 +3,7 @@ kind: ConfigMap
 metadata:
   name: streamlit-config
 data:
-  settings.json: |
+  settings-overrides.json: |
     {
-      "app-name": "OpenMS WebApp Template",
-      "online_deployment": true,
-      "enable_workspaces": true,
-      "workspaces_dir": "..",
-      "queue_settings": {
-        "default_timeout": 7200,
-        "result_ttl": 86400
-      },
-      "demo_workspaces": {
-        "enabled": true,
-        "source_dirs": ["example-data/workspaces"]
-      },
-      "max_threads": {
-        "local": 4,
-        "online": 2
-      },
-      "analytics": {
-        "matomo": {
-          "enabled": true,
-          "url": "https://cdn.matomo.cloud/openms.matomo.cloud",
-          "tag": "yDGK8bfY"
-        },
-        "google-analytics": {
-          "enabled": false,
-          "tag": ""
-        },
-        "piwik-pro": {
-          "enabled": false,
-          "tag": ""
-        }
-      }
+      "online_deployment": true
     }
diff --git a/k8s/base/kustomization.yaml b/k8s/base/kustomization.yaml
@@ -12,4 +12,5 @@ resources:
   - streamlit-service.yaml
   - rq-worker-deployment.yaml
   - ingress.yaml
+  - traefik-ingressroute.yaml
   - cleanup-cronjob.yaml
diff --git a/k8s/base/rq-worker-deployment.yaml b/k8s/base/rq-worker-deployment.yaml
@@ -32,7 +32,9 @@ spec:
           command: ["/bin/bash", "-c"]
           args:
             - |
+              set -euo pipefail
               source /root/miniforge3/bin/activate streamlit-env
+              jq -s '.[0] * .[1]' /app/settings.json /app/settings-overrides.json > /tmp/settings-merged.json && mv /tmp/settings-merged.json /app/settings.json
               exec rq worker openms-workflows --url $REDIS_URL
           env:
             - name: REDIS_URL
@@ -41,8 +43,8 @@ spec:
             - name: workspaces
               mountPath: /workspaces-streamlit-template
             - name: config
-              mountPath: /app/settings.json
-              subPath: settings.json
+              mountPath: /app/settings-overrides.json
+              subPath: settings-overrides.json
               readOnly: true
           resources:
             requests:
diff --git a/k8s/base/streamlit-deployment.yaml b/k8s/base/streamlit-deployment.yaml
@@ -32,7 +32,9 @@ spec:
           command: ["/bin/bash", "-c"]
           args:
             - |
+              set -euo pipefail
               source /root/miniforge3/bin/activate streamlit-env
+              jq -s '.[0] * .[1]' /app/settings.json /app/settings-overrides.json > /tmp/settings-merged.json && mv /tmp/settings-merged.json /app/settings.json
               exec streamlit run app.py --server.address 0.0.0.0
           ports:
             - containerPort: 8501
@@ -43,8 +45,8 @@ spec:
             - name: workspaces
               mountPath: /workspaces-streamlit-template
             - name: config
-              mountPath: /app/settings.json
-              subPath: settings.json
+              mountPath: /app/settings-overrides.json
+              subPath: settings-overrides.json
               readOnly: true
           readinessProbe:
             httpGet:
diff --git a/k8s/base/traefik-ingressroute.yaml b/k8s/base/traefik-ingressroute.yaml
@@ -0,0 +1,18 @@
+apiVersion: traefik.io/v1alpha1
+kind: IngressRoute
+metadata:
+  name: streamlit-traefik
+spec:
+  entryPoints:
+    - web
+  routes:
+    - match: PathPrefix(`/`)
+      kind: Rule
+      services:
+        - name: streamlit
+          port: 8501
+          sticky:
+            cookie:
+              name: stroute
+              httpOnly: true
+              sameSite: lax
diff --git a/k8s/overlays/template-app/kustomization.yaml b/k8s/overlays/template-app/kustomization.yaml
@@ -22,3 +22,24 @@ patches:
       - op: replace
         path: /spec/rules/0/host
         value: template.openms.example.de
+  - target:
+      kind: Deployment
+      name: streamlit
+    patch: |
+      - op: replace
+        path: /spec/template/spec/containers/0/env/0/value
+        value: "redis://template-app-redis:6379/0"
+  - target:
+      kind: Deployment
+      name: rq-worker
+    patch: |
+      - op: replace
+        path: /spec/template/spec/containers/0/env/0/value
+        value: "redis://template-app-redis:6379/0"
+  - target:
+      kind: IngressRoute
+      name: streamlit-traefik
+    patch: |
+      - op: replace
+        path: /spec/routes/0/services/0/name
+        value: "template-app-streamlit"
