Refactor node diagnostics into gather_nodes, adopt PR kubevirt#241

Merge gather_node_diagnostics into gather_nodes to eliminate the double
DaemonSet deployment that occurred when both scripts ran sequentially in
the default gather pipeline. The merged script collects all per-node
data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel
diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle
with one privileged pod per node (~25s/node).

Key changes:

- Merge: absorb all 24 diagnostic files from gather_node_diagnostics
  plus all networking/SR-IOV/CNI/audit files from old gather_nodes into
  a single collect_node() function. Remove gather_node_diagnostics.

- Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES
  cap (default 5) using background subshells. Within each node, oc exec
  calls are batched ~7 at a time.

- Remove oc debug fallback: the slow path (~480s/node) that spawned
  individual debug pods is gone. If no DaemonSet pod is available for a
  node, it logs a warning and skips.

- Fix DaemonSet wait: check numberReady + numberUnavailable >= desired
  instead of numberReady == desired, so unschedulable nodes don't cause
  a 300s wait for pods that will never start.

- Timeout guards: every oc command is wrapped in timeout. The old
  gather_nodes had ~22 unprotected oc exec/get/apply/delete calls.

- Namespace safety: check for stuck Terminating namespace before
  deploying; cleanup uses --wait=false to avoid blocking on namespace
  finalization.

- Size caps: audit.log capped at 50 MB (skipped entirely in incident
  mode), QEMU log tar transfer capped at 50 MB, curl inside
  gather_prometheus_instant gets --connect-timeout/--max-time to prevent
  orphan processes.

Adopt 4 collection scripts from PR kubevirt#241 for the default gather:
- gather_clusterroles: ClusterRole RBAC baseline
- gather_cnv_events: GuestPanicked/LivenessProbeFailed events
- gather_prometheus_instant: 14 instant Prometheus health queries
- gather_windows_nodes: Windows worker node posture

Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper
providing output directory conventions and NODES list.

Protect every oc command across all new scripts with timeout guards. Add
a static analysis test that verifies all oc commands in collection
scripts are timeout-protected, plus presence checks for each new
script's output.

This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and
gather_worker_kernel_log are absorbed into gather_nodes, the other 4
scripts adopted directly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>

Author: tiraboschi

README.md

@@ -15,6 +15,11 @@ You will get a dump of:
 - The Hyperconverged Cluster Operator namespaces (and its children objects)
 - All namespaces (and their children objects) that belong to any KubeVirt resources
 - All KubeVirt CRD's definitions
+- Per-node system and storage diagnostics (dmesg, dmidecode, sysctl, NFS config, PSI counters, kernel red-flag scan)
+- CNV guest events (GuestPanicked, LivenessProbeFailed) across VM namespaces
+- Prometheus instant metrics (cluster utilization, VM phases, storage latencies, NFS counters)
+- ClusterRole definitions (cluster-reader posture, KubeVirt RBAC baseline)
+- Windows worker node posture data (when Windows nodes are present)
 
 By default, the VMs definitions won't be included, but only the VM Instances' custom resources.
 
@@ -69,11 +74,28 @@ Usage: oc adm must-gather --image=quay.io/kubevirt/must-gather -- /usr/bin/gathe
   > - ssp
   > - virtualmachines
   > - webhooks
+  > - instancetypes
+  > - virtualization
+  > - cnv_events
+  > - prometheus_instant
+  > - clusterroles
+  > - windows_nodes
 
   > You can also choose to enable optional collectors combining one
   > or more of the following parameters:
   --images
   --vms_details
+
+  > Incident collection for a specific VM at a known time.
+  > Unlike a full must-gather, this collects ONLY data pertinent to the
+  > incident: right VM, right node, right time window. No cluster-wide
+  > noise. Timeboxed to 10 minutes.
+  --vm-incident --incident-time=<RFC3339 timestamp>
+    Requires NS and VM environment variables. Skips all other collectors.
+    Example:
+      oc adm must-gather --image=quay.io/kubevirt/must-gather \
+        -- NS=myns VM=myvm \
+        /usr/bin/gather --vm-incident --incident-time=2026-06-06T06:06:00Z
 ```
 
 ### Parallelism

collection-scripts/common.sh

@@ -5,6 +5,9 @@ export BASE_COLLECTION_PATH="${BASE_COLLECTION_PATH:-/must-gather}"
 export PROS=${PROS:-5}
 export INSTALLATION_NAMESPACE=${INSTALLATION_NAMESPACE:-kubevirt-hyperconverged}
 
+# Shared kernel red-flag grep pattern — used by gather_nodes and gather_vm_incident
+export KERNEL_REDFLAG_PATTERN='nfs:|NFS |nfs |NFSERR|server not responding|zero writ|call_transmit|cb path stale|not responding for|qemu|QEMU|Out of memory|oom-kill|Killed process.*qemu|task .* blocked|blocked for more than|stuck for|jiffies'
+
 function check_command {
     if [[ -z "$USR_BIN_GATHER" ]]; then
         echo "This script should not be directly executed." 1>&2

collection-scripts/gather

@@ -19,6 +19,10 @@ function main() {
     "webhooks"
     "instancetypes"
     "virtualization"
+    "cnv_events"
+    "prometheus_instant"
+    "clusterroles"
+    "windows_nodes"
   )
   declare requested_scripts=("${mandatory_scripts[@]}")
   VM_INCIDENT=false

collection-scripts/gather_clusterroles

@@ -0,0 +1,47 @@
+#!/bin/bash
+
+# ClusterRole definitions for cluster-reader posture (CNV / KubeVirt RBAC baseline).
+# Output: cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/
+
+# shellcheck disable=SC2034  # COLLECTION is used by lib_env after sourcing
+COLLECTION=cluster-scoped-resources/rbac.authorization.k8s.io
+
+# shellcheck source=lib_env
+. /usr/bin/lib_env
+
+mkdir -p "${DIR}/aggregated"
+
+echo "INFO-CNV: clusterrole cluster-reader"
+timeout 30 oc get clusterrole cluster-reader -o yaml > "${DIR}/cluster-reader.yaml" 2>&1 || true
+
+echo "INFO-CNV: clusterroles aggregated to cluster-reader"
+mapfile -t AGG_ROLES < <(timeout 30 oc get clusterroles -l rbac.authorization.k8s.io/aggregate-to-cluster-reader=true -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null | sort -u)
+printf '%s\n' "${AGG_ROLES[@]}" > "${DIR}/aggregated_role_names.txt"
+for role in "${AGG_ROLES[@]}"; do
+	[[ -z "${role}" ]] && continue
+	echo "INFO-CNV: aggregated clusterrole ${role}"
+	timeout 30 oc get clusterrole "${role}" -o yaml > "${DIR}/aggregated/${role}.yaml" 2>&1 || true
+done
+
+echo "INFO-CNV: CNV/kubevirt-related clusterroles"
+mapfile -t CNV_ROLES < <(timeout 30 oc get clusterroles -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null | \
+	grep -iE 'kubevirt|hyperconverged|cnv|virt-api|virt-controller|virt-handler|hostpath-provisioner|kubevirt-hyperconverged' | sort -u)
+printf '%s\n' "${CNV_ROLES[@]}" > "${DIR}/cnv_related_role_names.txt"
+mkdir -p "${DIR}/cnv_related"
+for role in "${CNV_ROLES[@]}"; do
+	[[ -z "${role}" ]] && continue
+	echo "INFO-CNV: cnv clusterrole ${role}"
+	timeout 30 oc get clusterrole "${role}" -o yaml > "${DIR}/cnv_related/${role}.yaml" 2>&1 || true
+done
+
+echo "INFO-CNV: clusterrolebindings referencing cluster-reader"
+timeout 30 oc get clusterrolebindings -o json 2>/dev/null | jq -r '
+  .items[]
+  | select(.roleRef.name == "cluster-reader")
+  | [.metadata.name, (.subjects[]? | (.kind + "/" + .name + (if .namespace then "@" + .namespace else "" end)))]
+  | @tsv
+' > "${DIR}/cluster_reader_binding_subjects.tsv" 2>/dev/null || true
+
+date -u +"%Y-%m-%dT%H:%M:%SZ" > "${DIR}/timestamp"
+
+sync

collection-scripts/gather_cnv_events

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# Collect GuestPanicked and virt-launcher LivenessProbeFailed events from VM namespaces.
+# Output: workload-scoped-resources/cnv_events/
+
+# shellcheck disable=SC2034  # COLLECTION is used by lib_env after sourcing
+COLLECTION=workload-scoped-resources
+
+# shellcheck source=lib_env
+. /usr/bin/lib_env
+
+if ! timeout 10 oc get ns "${INSTALLATION_NAMESPACE}" &>/dev/null; then
+	echo "INFO-CNV: namespace ${INSTALLATION_NAMESPACE} not found, skipping CNV guest events"
+	exit 0
+fi
+
+declare -A NS_SEEN=()
+
+_add_ns() {
+	local ns="$1"
+	[[ -z "${ns}" || "${ns}" == "null" ]] && return
+	NS_SEEN["${ns}"]=1
+}
+
+while IFS= read -r ns; do
+	_add_ns "${ns}"
+done < <(timeout 30 oc get vmi -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' 2>/dev/null | sort -u)
+
+while IFS= read -r ns; do
+	_add_ns "${ns}"
+done < <(timeout 30 oc get vm -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' 2>/dev/null | sort -u)
+
+while IFS= read -r ns; do
+	_add_ns "${ns}"
+done < <(timeout 60 oc get pods -A -o json 2>/dev/null | jq -r '.items[] | select(.metadata.name | startswith("virt-launcher-")) | .metadata.namespace' | sort -u)
+
+_add_ns "${INSTALLATION_NAMESPACE}"
+
+if [[ ${#NS_SEEN[@]} -eq 0 ]]; then
+	echo "INFO-CNV: no VM namespaces discovered, collecting ${INSTALLATION_NAMESPACE} events only"
+	NS_SEEN["${INSTALLATION_NAMESPACE}"]=1
+fi
+
+printf '%s\n' "${!NS_SEEN[@]}" | sort -u > "${DIR}/audited_namespaces.txt"
+
+while IFS= read -r ns; do
+	[[ -z "${ns}" ]] && continue
+	echo "INFO-CNV: CNV events - namespace ${ns}"
+	timeout 30 oc get events -n "${ns}" \
+		--field-selector reason=GuestPanicked \
+		-o yaml > "${DIR}/${ns}_GuestPanicked.yaml" 2>&1 || true
+
+	timeout 30 oc get events -n "${ns}" \
+		--field-selector reason=LivenessProbeFailed \
+		-o yaml > "${DIR}/${ns}_LivenessProbeFailed.yaml" 2>&1 || true
+done < <(sort -u "${DIR}/audited_namespaces.txt")
+
+timeout 30 oc get events -A --field-selector reason=GuestPanicked \
+	-o yaml > "${DIR}/all_GuestPanicked.yaml" 2>/dev/null || true
+
+timeout 30 oc get events -A --field-selector reason=LivenessProbeFailed -o wide 2>/dev/null | \
+	{ head -1; grep -E 'virt-launcher|VirtLauncher' || true; } \
+	> "${DIR}/all_LivenessProbeFailed_virt_launcher_table" || true
+
+date -u +"%Y-%m-%dT%H:%M:%SZ" > "${DIR}/timestamp"
+
+sync