kubectl kudo init failed on OpenShift 4.10.13

[dove-young]

What happened:
kubectl kudo init failed

What you expected to happen:

kubectl kudo init should be successful

How to reproduce it (as minimally and precisely as possible):

• download and install kudo v0.19.0
• run kubectl kudo init

 [root@st2 go]# kubectl kudo init
$KUDO_HOME has been configured at /root/.kudo
I0531 00:50:08.821073 1053584 request.go:645] Throttling request took 1.103939302s, request: GET:https://api.b-10.example.com:6443/apis/agent.open-cluster-management.io/v1?timeout=32s
Warnings
unable to find cert-manager deployment. Make sure cert-manager is running.
✅ installed crds
✅ installed namespace
✅ installed service account
Error: error installing: webhook: the server could not find the requested resource
[root@st2 go]#

• retry with trace kubectl kudo init -v 9

[root@st2 go]# kubectl kudo init -v 9
init cmd &{0xc00000e018 0xc00000e020 0x29722d8  Always false   kudo-system  false 300 false false false false /root/.kudo <nil> false}
/root/.kudo exists
/root/.kudo/repository exists
/root/.kudo/repository/repositories.yaml exists
$KUDO_HOME has been configured at /root/.kudo
create client
configuration from "/root/.kube/config" finds host https://api.b-10.example.com:6443
I0531 00:50:52.807179 1053628 request.go:645] Throttling request took 1.139566026s, request: GET:https://api.b-10.example.com:6443/apis/console.open-cluster-management.io/v1?timeout=32s
Run pre-install verify
Try to retrieve cert-manager CRD certificates.cert-manager.io
Got CRD. Group: cert-manager.io, Version: v1
Detected cert-manager CRDs /
Detected cert-manager cert-manager.io/v1
Errors
CRD operators.kudo.dev is already installed. Did you mean to use --upgrade?
Error: failed to verify installation requirements
[root@st2 go]#

• retry with upgrade kubectl kudo init -v 9 --upgrade

[root@st2 go]# kubectl kudo init -v 9 --upgrade
init cmd &{0xc00012e008 0xc00012e010 0x29722d8  Always false   kudo-system  false 300 false false true false /root/.kudo <nil> false}
/root/.kudo exists
/root/.kudo/repository exists
/root/.kudo/repository/repositories.yaml exists
$KUDO_HOME has been configured at /root/.kudo
create client
configuration from "/root/.kube/config" finds host https://api.b-10.example.com:6443
I0531 00:51:35.285933 1053657 request.go:645] Throttling request took 1.135512929s, request: GET:https://api.b-10.example.com:6443/apis/inventory.open-cluster-management.io/v1alpha1?timeout=32s
Run pre-upgrade verify
Service Account kudo-system/kudo-manager exists
Service Account kudo-system/kudo-manager has cluster-admin role
Try to retrieve cert-manager CRD certificates.cert-manager.io
Got CRD. Group: cert-manager.io, Version: v1
Detected cert-manager CRDs /
Detected cert-manager cert-manager.io/v1
CRD certificates.cert-manager.io is served with version v1
CRD issuers.cert-manager.io is served with version v1
Verify that 0 required migrations can be applied
Warnings
unable to find cert-manager deployment. Make sure cert-manager is running.
Upgrade KUDO
Uninstall KUDO manager stateful set
Deleting obj kudo-system/kudo-controller-manager is already NotFound, return now
Uninstall webhook
I0531 00:51:46.283975 1053657 request.go:645] Throttling request took 1.197148932s, request: GET:https://api.b-10.example.com:6443/api/v1?timeout=32s
I0531 00:51:56.484040 1053657 request.go:645] Throttling request took 11.397069371s, request: GET:https://api.b-10.example.com:6443/apis/consoleui.management.ibm.com/v1alpha1?timeout=32s
I0531 00:52:06.681489 1053657 request.go:645] Throttling request took 21.59444326s, request: GET:https://api.b-10.example.com:6443/apis/imageregistry.open-cluster-management.io/v1alpha1?timeout=32s
error upgrading: failed to uninstall webhook: failed to uninstall WebHook: failed to delete /kudo-manager-instance-admission-webhook-config: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
Error: error upgrading: failed to uninstall webhook: failed to uninstall WebHook: failed to delete /kudo-manager-instance-admission-webhook-config: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
[root@st2 go]#

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2-0-g52c56ce", GitCommit:"6082e941e6d62f3a0c6ca8ba52927100948b1d0d", GitTreeState:"clean", BuildDate:"2020-11-14T01:08:04Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5+b463d71", GitCommit:"3c28e7a79b58e78b4c1dc1ab7e5f6c6c2d3aedd3", GitTreeState:"clean", BuildDate:"2022-04-27T00:35:47Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

• Kudo version (use kubectl kudo version):

KUDO Version: version.Info{GitVersion:"0.19.0", GitCommit:"4173395f", BuildDate:"2021-04-27T15:30:48Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64", KubernetesClientVersion:"v0.19.2"}

• Operator:
• operatorVersion:
• Cloud provider or hardware configuration: OpenShift 4.10.13
• OS (e.g. from /etc/os-release):

NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.5"

• Kernel (e.g. uname -a):

Linux st2.example.com 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
• Others:

[dove-young]

I tried the dry-run yaml one by one, and the failure is caused by apiVersion: admissionregistration.k8s.io/v1beta1

[root@st2 KUDO]# cat <<EOF | kubectl apply -f -
> apiVersion: admissionregistration.k8s.io/v1beta1
> kind: MutatingWebhookConfiguration
> metadata:
>   annotations:
>     cert-manager.io/inject-ca-from: kudo-system/kudo-webhook-server-certificate
>   creationTimestamp: null
>   name: kudo-manager-instance-admission-webhook-config
> webhooks:
> - clientConfig:
>     service:
>       name: kudo-controller-manager-service
>       namespace: kudo-system
>       path: /admit-kudo-dev-v1beta1-instance
>   failurePolicy: Fail
>   matchPolicy: Equivalent
>   name: instance-admission.kudo.dev
>   rules:
>   - apiGroups:
>     - kudo.dev
>     apiVersions:
>     - v1beta1
>     operations:
>     - CREATE
>     - UPDATE
>     resources:
>     - instances
>     scope: Namespaced
>   sideEffects: None
> EOF
I0531 01:32:21.709267 1057611 request.go:621] Throttling request took 1.180783366s, request: GET:https://api.b-10.example.com:6443/apis/lighthouse.submariner.io/v2alpha1?timeout=32s
error: unable to recognize "STDIN": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

[dove-young]

When I changed it to apiVersion: admissionregistration.k8s.io/v1, it works

[root@st2 KUDO]# cat <<EOF | kubectl apply -f -
> apiVersion: admissionregistration.k8s.io/v1
> kind: MutatingWebhookConfiguration
> metadata:
>   annotations:
>     cert-manager.io/inject-ca-from: kudo-system/kudo-webhook-server-certificate
>   creationTimestamp: null
>   name: kudo-manager-instance-admission-webhook-config
> webhooks:
> - clientConfig:
>     service:
>       name: kudo-controller-manager-service
>       namespace: kudo-system
>       path: /admit-kudo-dev-v1beta1-instance
>   failurePolicy: Fail
>   matchPolicy: Equivalent
>   name: instance-admission.kudo.dev
>   rules:
>   - apiGroups:
>     - kudo.dev
>     apiVersions:
>     - v1beta1
>     operations:
>     - CREATE
>     - UPDATE
>     resources:
>     - instances
>     scope: Namespaced
>   sideEffects: None
>   admissionReviewVersions: ["v1", "v1beta1"]
> EOF
I0531 01:43:25.570267 1060221 request.go:621] Throttling request took 1.181808496s, request: GET:https://api.b-10.example.com:6443/apis/view.open-cluster-management.io/v1beta1?timeout=32s
mutatingwebhookconfiguration.admissionregistration.k8s.io/kudo-manager-instance-admission-webhook-config created
[root@st2 KUDO]#