Good morning,
we're currently building several bare-metal kubernetes clusters, intended to host a larger fleet of k3k clusters (at the moment in shared mode). One of the planned workloads will be kubevirt instances. Leaving the other found issues with kubevirt in virtual clusters aside, one feature will be necessary to ensure smooth operation and maintainability: Propagation of node labels, roles and state.
Roles and labels are required to schedule virtual machines inside virtual clusters on the correct physical hosts. Our control plane runs on less capable hardware than the worker nodes, virtual machines should only be scheduled on the worker nodes. That requires role and/or label synchronization.
State sync is required to allow PodDisruptionBudgets to block virtual machines from being evacuated, while a VMs on a node in "cordoned" state are being live-migrated into a new pod on a different host. Right now any workload in a shared cluster would be surprised by the host rebooting for maintenance, for example due to node-rotation or updates being applied. Workloads inside a shared k3k cluster should have the same options available to react to cordoned hosts, as well as draining a host node should equally wait for pods in the virtual cluster to get terminated gracefully.
In the current state we'd be forced to run workloads depending on this outside of virtual clusters. This doesn't only affect virtual machines, but basically everything that forms a cluster, like the cnpg (postgres) operator.
We can contribute that functionality if that helps. (For transparency: We're using LLMs.)
Good morning,
we're currently building several bare-metal kubernetes clusters, intended to host a larger fleet of k3k clusters (at the moment in shared mode). One of the planned workloads will be kubevirt instances. Leaving the other found issues with kubevirt in virtual clusters aside, one feature will be necessary to ensure smooth operation and maintainability: Propagation of node labels, roles and state.
Roles and labels are required to schedule virtual machines inside virtual clusters on the correct physical hosts. Our control plane runs on less capable hardware than the worker nodes, virtual machines should only be scheduled on the worker nodes. That requires role and/or label synchronization.
State sync is required to allow PodDisruptionBudgets to block virtual machines from being evacuated, while a VMs on a node in "cordoned" state are being live-migrated into a new pod on a different host. Right now any workload in a shared cluster would be surprised by the host rebooting for maintenance, for example due to node-rotation or updates being applied. Workloads inside a shared k3k cluster should have the same options available to react to cordoned hosts, as well as draining a host node should equally wait for pods in the virtual cluster to get terminated gracefully.
In the current state we'd be forced to run workloads depending on this outside of virtual clusters. This doesn't only affect virtual machines, but basically everything that forms a cluster, like the cnpg (postgres) operator.
We can contribute that functionality if that helps. (For transparency: We're using LLMs.)