[Feature] Add opt-in observability stack

[amine-lah]

Validation Checklist

• I confirm that this is a Kubeflow-related issue.
• I am reporting this in the appropriate repository.
• I have followed the Kubeflow installation guidelines.
• The issue report is detailed and includes version numbers where applicable.
• I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).
• This issue pertains to Kubeflow development.
• I am available to work on this issue.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Detailed Description

Hello,

I'm proposing an optional common/observability kustomize component that gives KubeFlow users a
ready-to-use monitoring foundation for those looking for a simple and quick observability setup.

The initial content focuses on GPU workloads (NVIDIA DCGM + AMD ROCm), since those are the
highest-value metrics for KubeFlow use cases, but the component is structured to grow with
additional dashboards and ServiceMonitors over time.

Motivation

KubeFlow manifests currently ship with no observability layer. Users who want visibility into
their cluster, like utilization, allocation, resource availability, must wire up the monitoring
stack themselves from scratch: Prometheus Operator, ServiceMonitor configuration, Grafana
dashboard provisioning. This is a non-trivial barrier, especially for teams new to the ecosystem.

This contribution closes that gap by providing a concrete starting point that works with
standard upstream tooling (kube-prometheus-stack, NVIDIA/AMD GPU Operators), requires no
vendor-specific configuration, and integrates naturally into the existing common/ + kustomize
pattern used by the rest of the manifests repo.

Prior art

We have been running this observability stack in production at CERN on a KubeFlow cluster with
mixed NVIDIA and AMD GPU nodes, and we would be happy to contribute it to the community as a
starting point. Happy to open a PR if there is interest.
Our reference architecture is described here: https://architecture.cncf.io/architectures/cern-scientific-computing/

Proposed potential solution

As a start, I'm proposing the following: A new directory common/observability/ following the same base/ + overlays/kubeflow/
pattern used by other common/ components.

Contents

ServiceMonitors (requires Prometheus Operator):

• nvidia-dcgm-service-monitor.yaml — scrapes the DCGM exporter deployed by the NVIDIA GPU Operator (gpu-operator namespace, service label app: nvidia-dcgm-exporter, port gpu-metrics)
• amd-gpu-service-monitor.yaml — scrapes the AMD ROCm device metrics exporter deployed by the AMD GPU Operator (kube-amd-gpu namespace, service label app.kubernetes.io/name: device-metrics-exporter, port metrics)
• kepler-service-monitor.yaml — scrapes Kepler for per-pod energy consumption metrics (CPU + GPU power draw)

Unlike the GPU exporters (which are deployed by their respective GPU Operators), Kepler is not bundled with any other operator, so the component would also include Kepler's deployment manifests as an opt-in sub-component.

Grafana dashboards (provisioned as ConfigMaps with grafana_dashboard: "1"):

• GPU Cluster Usage — cluster-wide GPU utilization, memory, and resource usage per node
• GPU Namespace Usage — per-namespace GPU allocation and usage breakdown
• GPU Availability & Allocation — allocation ratios per GPU type, time-to-acquire-GPU metrics, pending GPU session tracking

Steps to Reproduce

N/A

Screenshots or Videos (Optional)

No response

[github-actions]

Welcome to the Kubeflow Manifests Repository

Thanks for opening your first issue. We are trying to deal with your issue on a best-effort basis. without guarantees.
If you need commercial support instead please reach out to us.
Please make sure to follow our default issue template so maintainers can quickly understand and reproduce the problem instead of just closing the issue.
The template asks for:

• Environment details (Kubernetes version, OS, platform)
• Components involved
• Steps to reproduce
• Current versus expected behavior
• Properly formatted logs, screenshots or error messages

Community Resources:

• Join us on the CNCF Slack: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
• Attend our community meetings: https://www.kubeflow.org/docs/about/community/

Feel free to ask questions — we are here to help.

[juliusvonkohout]

Hello, yes, please start with Prometheus and grafana operators. Can you deploy to kubeflow-monitorig-system? And we need tests as for the other components. In grafana we can use one organization per customer namespace.

[abdullahpathan22]

Hello @juliusvonkohout, I have raised a PR #3427 addressing this issue. Please review it. Thank you!

Observability Component – Implementation Summary

Architecture

The common/observability component follows the canonical base/ + overlays/kubeflow/ Kustomize pattern and is structured as an opt-in stack to avoid imposing overhead on clusters that don't require it.

---

ServiceMonitor Targeting

ServiceMonitor	Namespace	Selector	Port
nvidia-dcgm-service-monitor	gpu-operator	app: nvidia-dcgm-exporter	gpu-metrics
amd-gpu-service-monitor	kube-amd-gpu	app.kubernetes.io/name: device-metrics-exporter	metrics
kepler-service-monitor	kepler	—	http

All ServiceMonitors use cross-namespace scraping via explicit namespaceSelector, ensuring Prometheus can target workloads outside its own namespace without overly broad permissions.

---

Kepler Isolation

Kepler is deployed into its own dedicated kepler namespace with a privileged Pod Security Standard (PSS) admission label, deliberately isolated from the main stack which enforces a restricted PSS posture. This prevents Kepler's eBPF-based host instrumentation requirements from polluting or downgrading the security posture of the broader Kubeflow namespace hierarchy.

---

Grafana Dashboards

Three dashboards are provisioned — Cluster Usage, Namespace Usage, and Availability & Allocation — deployed as GrafanaDashboard CRs (Grafana Operator CRD) rather than raw ConfigMaps. This enables native reconciliation via the operator's controller loop, removing the dependency on the Grafana sidecar provisioner and providing drift detection out of the box.

---

Security & Hardening (Review Feedback Addressed)

• Credentials: Removed hardcoded admin/admin Grafana credentials. Credentials are now sourced from a Kubernetes Secret, decoupling sensitive values from manifests and enabling external secret management (e.g., ESO, Vault).
• RBAC Scoping: Grafana Operator ClusterRole has been scoped down — cluster-wide mutation permissions removed. Operator now holds only the minimum required verbs on targeted resource groups.
• Naming Collision Prevention: kepler-role renamed to kubeflow-kepler-role to avoid collision with any identically named ClusterRole that may exist in other operator deployments on the same cluster.
• PSS Version Pinning: Removed pod-security.kubernetes.io/*-version: latest labels. Versions are now pinned explicitly so that Kubernetes API version upgrades don't silently alter enforcement behavior.
• CI Hardening: model_registry_test.sh has been hardened (errexit/pipefail guards, input validation). OWNERS file permissions corrected to align with project governance expectations.

---