BSOD: new collection scripts gathering info used by Phased Gates rules to detect BSOD issues - #241
Open
xmorano2 wants to merge 8 commits into
Open
BSOD: new collection scripts gathering info used by Phased Gates rules to detect BSOD issues#241xmorano2 wants to merge 8 commits into
xmorano2 wants to merge 8 commits into
Conversation
…nts from VM namespaces
…d boot config per node. Read-only collection: cat/sysctl/status only — no modprobe, no writes, no cordon/drain/reboot
…MU OOM, stalled tasks)
kubevirt-prow
Bot
added
release-note-none
|
Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits. 📝 Please follow instructions in the contributing guide to update your commits with the DCO Full details of the Developer Certificate of Origin can be found at developercertificate.org. The list of commits missing DCO signoff:
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xmorano2 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @xmorano2. Thanks for your PR. I'm waiting for a kubevirt member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
openshift-ci
Bot
added
the
needs-ok-to-test
7 tasks
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 3, 2026
Extract the 130-line inline node diagnostics block from gather_vm_incident into a standalone gather_node_diagnostics script that supports dual-mode execution: - NODE_GATHER_POD set: fast oc exec against pre-deployed DaemonSet pod (incident mode, same batched pattern as before) - NODE_GATHER_POD unset: oc debug node/ with all commands batched into a single debug pod per node using delimiter-based output splitting (default gather mode, 2 debug pods per node instead of 8+) The new script collects 24 per-node files (the original 20 plus 4 NFS extras from PR kubevirt#241: nfs_sysctl, nfs_sysfs, nfs_slot_tuning_service, nfs_mountstats_delays) and cluster-level NFS MachineConfig audit. Add shared KERNEL_REDFLAG_PATTERN to common.sh so both gather_node_diagnostics and tests use the same grep pattern. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus queries - gather_windows_nodes: Windows worker node posture Add lib_env shared helper providing output directory conventions, NODES list, and check_command guard for the adopted scripts. This fully covers PR kubevirt#241's functionality (gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_node_diagnostics, the other 4 scripts adopted directly). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 4, 2026
Extract the 130-line inline node diagnostics block from gather_vm_incident into a standalone gather_node_diagnostics script with three execution modes: 1. NODE_GATHER_POD set: oc exec against caller-provided pod (incident mode) 2. NODE_GATHER_POD unset: deploy own DaemonSet, oc exec per-node pod (default gather — ~25s/node instead of ~480s/node with oc debug) 3. DaemonSet deployment fails: oc debug node/ fallback with all commands batched into a single debug pod using delimiter-based output splitting The script collects 24 per-node files (the original 20 plus nfs_sysctl, nfs_sysfs, nfs_slot_tuning_service, nfs_mountstats_delays from PR kubevirt#241) and cluster-level NFS MachineConfig audit under cluster-scoped-resources/. Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Protect every oc command across all new scripts with timeout guards to prevent hangs. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_node_diagnostics, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 4, 2026
Extract the 130-line inline node diagnostics block from gather_vm_incident into a standalone gather_node_diagnostics script with three execution modes: 1. NODE_GATHER_POD set: oc exec against caller-provided pod (incident mode) 2. NODE_GATHER_POD unset: deploy own DaemonSet, oc exec per-node pod (default gather — ~25s/node instead of ~480s/node with oc debug) 3. DaemonSet deployment fails: oc debug node/ fallback with all commands batched into a single debug pod using delimiter-based output splitting The script collects 24 per-node files (the original 20 plus nfs_sysctl, nfs_sysfs, nfs_slot_tuning_service, nfs_mountstats_delays from PR kubevirt#241) and cluster-level NFS MachineConfig audit under cluster-scoped-resources/. Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Protect every oc command across all new scripts with timeout guards to prevent hangs. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_node_diagnostics, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 5, 2026
Extract the 130-line inline node diagnostics block from gather_vm_incident into a standalone gather_node_diagnostics script with three execution modes: 1. NODE_GATHER_POD set: oc exec against caller-provided pod (incident mode) 2. NODE_GATHER_POD unset: deploy own DaemonSet, oc exec per-node pod (default gather — ~25s/node instead of ~480s/node with oc debug) 3. DaemonSet deployment fails: oc debug node/ fallback with all commands batched into a single debug pod using delimiter-based output splitting The script collects 24 per-node files (the original 20 plus nfs_sysctl, nfs_sysfs, nfs_slot_tuning_service, nfs_mountstats_delays from PR kubevirt#241) and cluster-level NFS MachineConfig audit under cluster-scoped-resources/. Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Protect every oc command across all new scripts with timeout guards to prevent hangs. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_node_diagnostics, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 6, 2026
Merge gather_node_diagnostics into gather_nodes to eliminate the double DaemonSet deployment that occurred when both scripts ran sequentially in the default gather pipeline. The merged script collects all per-node data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle with one privileged pod per node (~25s/node). Key changes: - Merge: absorb all 24 diagnostic files from gather_node_diagnostics plus all networking/SR-IOV/CNI/audit files from old gather_nodes into a single collect_node() function. Remove gather_node_diagnostics. - Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES cap (default 5) using background subshells. Within each node, oc exec calls are batched ~7 at a time. - Remove oc debug fallback: the slow path (~480s/node) that spawned individual debug pods is gone. If no DaemonSet pod is available for a node, it logs a warning and skips. - Fix DaemonSet wait: check numberReady + numberUnavailable >= desired instead of numberReady == desired, so unschedulable nodes don't cause a 300s wait for pods that will never start. - Timeout guards: every oc command is wrapped in timeout. The old gather_nodes had ~22 unprotected oc exec/get/apply/delete calls. - Namespace safety: check for stuck Terminating namespace before deploying; cleanup uses --wait=false to avoid blocking on namespace finalization. - Size caps: audit.log capped at 50 MB (skipped entirely in incident mode), QEMU log tar transfer capped at 50 MB, curl inside gather_prometheus_instant gets --connect-timeout/--max-time to prevent orphan processes. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Protect every oc command across all new scripts with timeout guards. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_nodes, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 6, 2026
Merge gather_node_diagnostics into gather_nodes to eliminate the double DaemonSet deployment that occurred when both scripts ran sequentially in the default gather pipeline. The merged script collects all per-node data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle with one privileged pod per node (~25s/node). Key changes: - Merge: absorb all 24 diagnostic files from gather_node_diagnostics plus all networking/SR-IOV/CNI/audit files from old gather_nodes into a single collect_node() function. Remove gather_node_diagnostics. - Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES cap (default 5) using background subshells. Within each node, oc exec calls are batched ~7 at a time. - Remove oc debug fallback: the slow path (~480s/node) that spawned individual debug pods is gone. If no DaemonSet pod is available for a node, it logs a warning and skips. - Fix DaemonSet wait: check numberReady + numberUnavailable >= desired instead of numberReady == desired, so unschedulable nodes don't cause a 300s wait for pods that will never start. - Timeout guards: every oc command is wrapped in timeout. The old gather_nodes had ~22 unprotected oc exec/get/apply/delete calls. - Namespace safety: check for stuck Terminating namespace before deploying; cleanup uses --wait=false to avoid blocking on namespace finalization. - Size caps: audit.log capped at 50 MB (skipped entirely in incident mode), QEMU log tar transfer capped at 50 MB, curl inside gather_prometheus_instant gets --connect-timeout/--max-time to prevent orphan processes. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Protect every oc command across all new scripts with timeout guards. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_nodes, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 6, 2026
Merge gather_node_diagnostics into gather_nodes to eliminate the double DaemonSet deployment that occurred when both scripts ran sequentially in the default gather pipeline. The merged script collects all per-node data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle with one privileged pod per node (~25s/node). Key changes: - Merge: absorb all 24 diagnostic files from gather_node_diagnostics plus all networking/SR-IOV/CNI/audit files from old gather_nodes into a single collect_node() function. Remove gather_node_diagnostics. - Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES cap (default 5) using background subshells. Within each node, oc exec calls are batched ~7 at a time. - Remove oc debug fallback: the slow path (~480s/node) that spawned individual debug pods is gone. If no DaemonSet pod is available for a node, it logs a warning and skips. - Fix DaemonSet wait: check numberReady + numberUnavailable >= desired instead of numberReady == desired, so unschedulable nodes don't cause a 300s wait for pods that will never start. - Timeout guards: every oc command is wrapped in timeout. The old gather_nodes had ~22 unprotected oc exec/get/apply/delete calls. - Namespace safety: check for stuck Terminating namespace before deploying; cleanup uses --wait=false to avoid blocking on namespace finalization. - Size caps: audit.log capped at 50 MB (skipped entirely in incident mode), QEMU log tar transfer capped at 50 MB, curl inside gather_prometheus_instant gets --connect-timeout/--max-time to prevent orphan processes. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Protect every oc command across all new scripts with timeout guards. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_nodes, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 6, 2026
Merge gather_node_diagnostics into gather_nodes to eliminate the double DaemonSet deployment that occurred when both scripts ran sequentially in the default gather pipeline. The merged script collects all per-node data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle with one privileged pod per node (~25s/node). Key changes: - Merge: absorb all 24 diagnostic files from gather_node_diagnostics plus all networking/SR-IOV/CNI/audit files from old gather_nodes into a single collect_node() function. Remove gather_node_diagnostics. - Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES cap (default 5) using background subshells. Within each node, oc exec calls are batched ~7 at a time. - Remove oc debug fallback: the slow path (~480s/node) that spawned individual debug pods is gone. If no DaemonSet pod is available for a node, it logs a warning and skips. - Fix DaemonSet wait: check numberReady + numberUnavailable >= desired instead of numberReady == desired, so unschedulable nodes don't cause a 300s wait for pods that will never start. - Timeout guards: every oc command is wrapped in timeout. The old gather_nodes had ~22 unprotected oc exec/get/apply/delete calls. - Namespace safety: check for stuck Terminating namespace before deploying; cleanup uses --wait=false to avoid blocking on namespace finalization. - Size caps: audit.log capped at 50 MB (skipped entirely in incident mode), QEMU log tar transfer capped at 50 MB, curl inside gather_prometheus_instant gets --connect-timeout/--max-time to prevent orphan processes. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Protect every oc command across all new scripts with timeout guards. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_nodes, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 6, 2026
Merge gather_node_diagnostics into gather_nodes to eliminate the double DaemonSet deployment that occurred when both scripts ran sequentially in the default gather pipeline. The merged script collects all per-node data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle with one privileged pod per node (~25s/node). Key changes: - Merge: absorb all 24 diagnostic files from gather_node_diagnostics plus all networking/SR-IOV/CNI/audit files from old gather_nodes into a single collect_node() function. Remove gather_node_diagnostics. - Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES cap (default 5) using background subshells. Within each node, oc exec calls are batched ~7 at a time. - Remove oc debug fallback: the slow path (~480s/node) that spawned individual debug pods is gone. If no DaemonSet pod is available for a node, it logs a warning and skips. - Fix DaemonSet wait: check numberReady + numberUnavailable >= desired instead of numberReady == desired, so unschedulable nodes don't cause a 300s wait for pods that will never start. - Timeout guards: every oc command is wrapped in timeout. The old gather_nodes had ~22 unprotected oc exec/get/apply/delete calls. - Namespace safety: check for stuck Terminating namespace before deploying; cleanup uses --wait=false to avoid blocking on namespace finalization. - Size caps: audit.log capped at 50 MB (skipped entirely in incident mode), QEMU log tar transfer capped at 50 MB, curl inside gather_prometheus_instant gets --connect-timeout/--max-time to prevent orphan processes. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Protect every oc command across all new scripts with timeout guards. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_nodes, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 7, 2026
Merge gather_node_diagnostics into gather_nodes to eliminate the double DaemonSet deployment that occurred when both scripts ran sequentially in the default gather pipeline. The merged script collects all per-node data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle with one privileged pod per node (~25s/node). Key changes: - Merge: absorb all 24 diagnostic files from gather_node_diagnostics plus all networking/SR-IOV/CNI/audit files from old gather_nodes into a single collect_node() function. Remove gather_node_diagnostics. - Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES cap (default 5) using background subshells. Within each node, oc exec calls are batched ~7 at a time. - Remove oc debug fallback: the slow path (~480s/node) that spawned individual debug pods is gone. If no DaemonSet pod is available for a node, it logs a warning and skips. - Fix DaemonSet wait: check numberReady + numberUnavailable >= desired instead of numberReady == desired, so unschedulable nodes don't cause a 300s wait for pods that will never start. - Timeout guards: every oc command is wrapped in timeout. The old gather_nodes had ~22 unprotected oc exec/get/apply/delete calls. - Namespace safety: check for stuck Terminating namespace before deploying; cleanup uses --wait=false to avoid blocking on namespace finalization. - Size caps: audit.log capped at 50 MB (skipped entirely in incident mode), QEMU log tar transfer capped at 50 MB, curl inside gather_prometheus_instant gets --connect-timeout/--max-time to prevent orphan processes. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Protect every oc command across all new scripts with timeout guards. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_nodes, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
tiraboschi
added a commit
to tiraboschi/must-gather-1
that referenced
this pull request
Jul 7, 2026
Merge gather_node_diagnostics into gather_nodes to eliminate the double DaemonSet deployment that occurred when both scripts ran sequentially in the default gather pipeline. The merged script collects all per-node data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle with one privileged pod per node (~25s/node). Key changes: - Merge: absorb all 24 diagnostic files from gather_node_diagnostics plus all networking/SR-IOV/CNI/audit files from old gather_nodes into a single collect_node() function. Remove gather_node_diagnostics. - Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES cap (default 5) using background subshells. Within each node, oc exec calls are batched ~7 at a time. - Remove oc debug fallback: the slow path (~480s/node) that spawned individual debug pods is gone. If no DaemonSet pod is available for a node, it logs a warning and skips. - Fix DaemonSet wait: check numberReady + numberUnavailable >= desired instead of numberReady == desired, so unschedulable nodes don't cause a 300s wait for pods that will never start. - Timeout guards: every oc command is wrapped in timeout. The old gather_nodes had ~22 unprotected oc exec/get/apply/delete calls. - Namespace safety: check for stuck Terminating namespace before deploying; cleanup uses --wait=false to avoid blocking on namespace finalization. - Size caps: audit.log capped at 50 MB (skipped entirely in incident mode), QEMU log tar transfer capped at 50 MB, curl inside gather_prometheus_instant gets --connect-timeout/--max-time to prevent orphan processes. Adopt 4 collection scripts from PR kubevirt#241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Protect every oc command across all new scripts with timeout guards. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR kubevirt#241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_nodes, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
kubevirt-prow Bot
pushed a commit
that referenced
this pull request
Jul 7, 2026
* Add --vm-incident focused collection for VM incident diagnosis When a VM hits a BSOD, kernel panic, or I/O hang, the data needed for root-cause analysis is scattered across four separate tools: CNV must-gather for VM definitions and virsh state, OCP must-gather for node logs and cluster operators, sosreport for dmesg/sysctl/NFS config/PSI counters, and manual Prometheus queries for performance metrics over time. Collecting all four takes hours, produces gigabytes of cluster-wide data, and requires an engineer to manually correlate the right node, the right time window, and the right VM across all of them. Logs are often captured from "now" rather than from when the incident occurred, missing the relevant entries entirely. This adds a --vm-incident mode that replaces that workflow with a single command. Given a VM name, namespace, and incident timestamp, it automatically identifies the node (via Prometheus kubevirt_vmi_info, surviving live-migration), scopes all collection to the incident window (T-24h to T+2h), and produces one small, focused archive covering: - Node: journal, kubelet logs, dmesg, dmidecode, kernel red-flag scan - Host config: sysctl, NFS config/module params, tuned profile, chrony, network interfaces - Storage I/O: mountstats, diskstats, PSI pressure, df, mounts - Storage chain: PVCs, PVs, StorageClasses, DataVolumes, VolumeAttachments, namespace events - Cluster health: ClusterOperators, nodes, MachineConfigPools, top node, KubeVirt version, VMIs on the incident node at incident time (noisy-neighbor detection) - VM state: VM/VMI definitions, virt-launcher and virt-handler logs (current + previous) - Live VM state (only if VM has not restarted): virsh dumpxml/domstats/ domblklist/domblkerror/domjobinfo, serial console log, QEMU logs, QEMU process cgroup stats (memory.current/max/events, cpu.stat) - 26 Prometheus metrics (VM CPU/memory/network/disk, node CPU/memory/ PSI/load/disk-I/O, storage volume stats, KubeVirt alerts) exported in OpenMetrics format for backfill into local Prometheus or VictoriaMetrics Metric definitions live in a declarative registry (incident_metrics.conf) so adding a metric is a one-line change with no code modifications. Output uses standard must-gather paths for omc compatibility. An incident-summary.yaml lists exactly what was collected and skipped. The entire collection completes in under a minute on a typical cluster and is timeboxed to 10 minutes (configurable via INCIDENT_TIMEOUT). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com> * Refactor node diagnostics into gather_nodes, adopt PR #241 Merge gather_node_diagnostics into gather_nodes to eliminate the double DaemonSet deployment that occurred when both scripts ran sequentially in the default gather pipeline. The merged script collects all per-node data (networking, SR-IOV, CNI, VFIO, audit, NFS tuning, kernel diagnostics, storage I/O, PSI pressure) in a single DaemonSet lifecycle with one privileged pod per node (~25s/node). Key changes: - Merge: absorb all 24 diagnostic files from gather_node_diagnostics plus all networking/SR-IOV/CNI/audit files from old gather_nodes into a single collect_node() function. Remove gather_node_diagnostics. - Parallelize: nodes are processed concurrently with MAX_PARALLEL_NODES cap (default 5) using background subshells. Within each node, oc exec calls are batched ~7 at a time. - Remove oc debug fallback: the slow path (~480s/node) that spawned individual debug pods is gone. If no DaemonSet pod is available for a node, it logs a warning and skips. - Fix DaemonSet wait: check numberReady + numberUnavailable >= desired instead of numberReady == desired, so unschedulable nodes don't cause a 300s wait for pods that will never start. - Timeout guards: every oc command is wrapped in timeout. The old gather_nodes had ~22 unprotected oc exec/get/apply/delete calls. - Namespace safety: check for stuck Terminating namespace before deploying; cleanup uses --wait=false to avoid blocking on namespace finalization. - Size caps: audit.log capped at 50 MB (skipped entirely in incident mode), QEMU log tar transfer capped at 50 MB, curl inside gather_prometheus_instant gets --connect-timeout/--max-time to prevent orphan processes. Adopt 4 collection scripts from PR #241 for the default gather: - gather_clusterroles: ClusterRole RBAC baseline - gather_cnv_events: GuestPanicked/LivenessProbeFailed events - gather_prometheus_instant: 14 instant Prometheus health queries - gather_windows_nodes: Windows worker node posture Add shared KERNEL_REDFLAG_PATTERN to common.sh and lib_env shared helper providing output directory conventions and NODES list. Protect every oc command across all new scripts with timeout guards. Add a static analysis test that verifies all oc commands in collection scripts are timeout-protected, plus presence checks for each new script's output. This fully covers PR #241's functionality — gather_nfs_node_tuning and gather_worker_kernel_log are absorbed into gather_nodes, the other 4 scripts adopted directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simone Tiraboschi <stirabos@redhat.com> --------- Signed-off-by: Simone Tiraboschi <stirabos@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
kubevirt-prow
Bot
added
the
needs-rebase
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release note: