Tolerations not applied to the DcgmExporter DaemonSet

[NicolasGrosjeanProbayes]

What I have done

I have added a toleration to DcgmExporter that I found here but using the DcgmExporter definition from your repository.

apiVersion: cloudwatch.aws.amazon.com/v1alpha1
kind: DcgmExporter
metadata:
  name: dcgm-exporter
  namespace: amazon-cloudwatch
  labels:
    k8s-app: dcgm-exporter
    version: v1
spec:
  image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.3-3.3.1-ubuntu22.04
  nodeSelector:
    kubernetes.io/os: linux
  serviceAccount: dcgm-exporter-service-acct
  tolerations:
  - key: "nvidia.com/gpu"
    operator: "Exists"
    effect: "NoSchedule"
[...]

Problem

The toleration is missing in the generated DaemonSet.

Workaround

Our AWS Solutions Architect has found a workaround by adding manually the toleration to the DaemonSet.

kubectl patch daemonset dcgm-exporter -n amazon-cloudwatch --type='json' -p='[ { "op": "add", "path": "/spec/template/spec/tolerations", "value": [ { "key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule" } ] } ]'

[sky333999]

Hello @NicolasGrosjeanProbayes , could you please provide some steps to reproduce the issue and provide details on which version you are using etc.

[NicolasGrosjeanProbayes]

Hello,

1. I have created an EKS cluster (version 1.29), a node group with CPU instance and scaled to one node
2. I set environment variables AWS_CLUSTER_NAME and AWS_REGION
3. I downloaded manifests and set cluster name and region with the following commands

curl -L -o cert-manager.yaml https://github.com/cert-manager/cert-manager/releases/download/v1.15.1/cert-manager.yaml

curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-custom-resource-definitions.yaml > cwagent-custom-resource-definitions.yaml

curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/main/k8s-quickstart/cwagent-operator-rendered.yaml | sed 's/{{cluster_name}}/'${AWS_CLUSTER_NAME}'/g;s/{{region_name}}/'${AWS_REGION}'/g' > cwagent-operator-rendered.yaml

4. I edited cwagent-custom-resource-definitions.yaml to replace the DcgmExporter definition by the one from your repository
5. I edited cwagent-operator-rendered.yaml to add toleration like in my first post
6. I deployed the manifests with kubectl v1.30.0-eks-036c24b

kubectl apply -f manifests/cloudwatch/cert-manager.yaml
kubectl apply --server-side -f manifests/cloudwatch/cwagent-custom-resource-definitions.yaml
kubectl apply -f manifests/cloudwatch/cwagent-operator-rendered.yaml

7. When I look the dcgm-exporter daemonset, I don't see the toleration