May 14 18:10:28.013 E ns/openshift-ovn-kubernetes pod/ovnkube-master-vrq8z node/ip-10-0-156-106.us-west-1.compute.internal container/ovnkube-master container exited with code 2 (Error): master.go:83] Won leader election; in active mode\nI0514 18:08:42.415513       1 master.go:229] Starting cluster master\nI0514 18:09:43.665349       1 reflector.go:530] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Endpoints total 173 items received\n2024/05/14 18:09:51 rpc2: client protocol error: read tcp 10.0.156.106:60318->10.0.154.202:9642: read: connection reset by peer\n2024/05/14 18:09:51 ssl:10.0.154.202:9642,ssl:10.0.156.106:9642,ssl:10.0.241.240:9642 disconnected. Reconnecting ... \n2024/05/14 18:09:51 ssl:10.0.154.202:9642,ssl:10.0.156.106:9642,ssl:10.0.241.240:9642 reconnected after 0 retries.\nI0514 18:09:57.663899       1 reflector.go:530] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Node total 55 items received\nI0514 18:10:01.162297       1 reflector.go:530] github.com/openshift/ovn-kubernetes/go-controller/pkg/crd/egressip/v1/apis/informers/externalversions/factory.go:117: Watch close - *v1.EgressIP total 0 items received\n2024/05/14 18:10:27 rpc2: client protocol error: read tcp 10.0.156.106:46392->10.0.154.202:9641: read: connection reset by peer\nE0514 18:10:27.394925       1 master.go:213] Failed to enable logical datapath groups: read tcp 10.0.156.106:46392->10.0.154.202:9641: read: connection reset by peer\nE0514 18:10:27.394934       1 master.go:243] Failed to set timestamp: read tcp 10.0.156.106:46392->10.0.154.202:9641: read: connection reset by peer\n2024/05/14 18:10:27 ssl:10.0.154.202:9641,ssl:10.0.156.106:9641,ssl:10.0.241.240:9641 disconnected. Reconnecting ... \npanic: read tcp 10.0.156.106:46392->10.0.154.202:9641: read: connection reset by peer\n\ngoroutine 631 [running]:\ngithub.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*Controller).Start.func1(0x1e85320, 0xc000f22000)\n	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/master.go:94 +0x265\ncreated by k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run\n	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:207 +0x113\n
May 14 18:10:46.920 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-156-106.us-west-1.compute.internal node/ip-10-0-156-106.us-west-1.compute.internal container/kube-scheduler container exited with code 1 (Error):  node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0514 18:09:53.853857       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-oauth-apiserver/apiserver-8559c747c8-7582t" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0514 18:10:01.854237       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-etcd/etcd-quorum-guard-5485fcd469-m7xp8" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0514 18:10:03.854542       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-apiserver/apiserver-56f6b658cb-r5ws9" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0514 18:10:03.854953       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-oauth-apiserver/apiserver-8559c747c8-7582t" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nE0514 18:10:44.613068       1 leaderelection.go:325] error retrieving resource lock openshift-kube-scheduler/kube-scheduler: etcdserver: request timed out\nI0514 18:10:45.941259       1 leaderelection.go:278] failed to renew lease openshift-kube-scheduler/kube-scheduler: timed out waiting for the condition\nE0514 18:10:45.941386       1 leaderelection.go:301] Failed to release lock: resource name may not be empty\nF0514 18:10:45.941405       1 server.go:217] leaderelection lost\n
May 14 18:10:47.455 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-241-240.us-west-1.compute.internal node/ip-10-0-241-240.us-west-1.compute.internal container/kube-controller-manager container exited with code 1 (Error): is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use IPv4 as the IP Family based on familyOf(ClusterIP:172.30.244.82).\nE0514 18:10:44.603815       1 leaderelection.go:325] error retrieving resource lock kube-system/kube-controller-manager: etcdserver: request timed out\nI0514 18:10:45.406959       1 utils.go:424] couldn't find ipfamilies for headless service: openshift-monitoring/node-exporter likely because controller manager is likely connected to an old apiserver that does not support ip families yet. The service endpoint slice will use dual stack families until api-server default it correctly\nI0514 18:10:45.509481       1 utils.go:413] couldn't find ipfamilies for service: openshift-machine-config-operator/machine-config-daemon. This could happen if controller manager is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use IPv4 as the IP Family based on familyOf(ClusterIP:172.30.87.193).\nI0514 18:10:46.520883       1 utils.go:413] couldn't find ipfamilies for service: openshift-machine-config-operator/machine-config-daemon. This could happen if controller manager is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use IPv4 as the IP Family based on familyOf(ClusterIP:172.30.87.193).\nI0514 18:10:46.884756       1 utils.go:413] couldn't find ipfamilies for service: openshift-kube-scheduler/scheduler. This could happen if controller manager is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use IPv4 as the IP Family based on familyOf(ClusterIP:172.30.184.31).\nI0514 18:10:46.999095       1 leaderelection.go:278] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition\nE0514 18:10:46.999196       1 leaderelection.go:301] Failed to release lock: resource name may not be empty\nF0514 18:10:46.999296       1 controllermanager.go:320] leaderelection lost\n

May 14 18:13:56.223 E ns/openshift-cluster-storage-operator pod/csi-snapshot-controller-854f858fbf-ks9hh node/ip-10-0-154-202.us-west-1.compute.internal container/snapshot-controller container exited with code 255 (Error):
May 14 18:13:56.295 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-241-240.us-west-1.compute.internal node/ip-10-0-241-240.us-west-1.compute.internal container/kube-scheduler container exited with code 1 (Error):  node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0514 18:13:30.676258       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-apiserver/apiserver-56f6b658cb-j2m7c" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0514 18:13:30.676542       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-monitoring/prometheus-k8s-0" err="0/6 nodes are available: 2 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) had volume node affinity conflict, 2 node(s) were unschedulable."\nI0514 18:13:30.676747       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-monitoring/prometheus-k8s-1" err="0/6 nodes are available: 2 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) had volume node affinity conflict, 2 node(s) were unschedulable."\nI0514 18:13:30.676989       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-etcd/etcd-quorum-guard-5485fcd469-929cm" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nE0514 18:13:35.635772       1 leaderelection.go:325] error retrieving resource lock openshift-kube-scheduler/kube-scheduler: Get "https://api-int.ci-op-b10rgbvf-374d8.aws-2.ci.openshift.org:6443/api/v1/namespaces/openshift-kube-scheduler/configmaps/kube-scheduler?timeout=10s": context deadline exceeded\nI0514 18:13:35.635830       1 leaderelection.go:278] failed to renew lease openshift-kube-scheduler/kube-scheduler: timed out waiting for the condition\nE0514 18:13:35.635880       1 leaderelection.go:301] Failed to release lock: resource name may not be empty\nF0514 18:13:35.635904       1 server.go:217] leaderelection lost\n
May 14 18:14:02.897 E clusteroperator/machine-config changed Degraded to True: MachineConfigDaemonFailed: Unable to apply 4.7.0-0.ci-2024-05-14-162942: rpc error: code = Unavailable desc = transport is closing

May 14 18:16:20.574 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ip-10-0-225-74.us-west-1.compute.internal container/prometheus container exited with code 2 (Error): level=error ts=2024-05-14T18:16:19.492Z caller=main.go:289 msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" err="open /etc/prometheus/config_out/prometheus.env.yaml: no such file or directory"\n
May 14 18:16:34.618 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-154-202.us-west-1.compute.internal node/ip-10-0-154-202.us-west-1.compute.internal container/kube-scheduler container exited with code 1 (Error): h Pod's node affinity."\nI0514 18:16:02.665354       1 scheduler.go:606] "Successfully bound pod to node" pod="openshift-monitoring/prometheus-k8s-0" node="ip-10-0-225-74.us-west-1.compute.internal" evaluatedNodes=6 feasibleNodes=1\nI0514 18:16:02.665640       1 scheduler.go:606] "Successfully bound pod to node" pod="openshift-monitoring/prometheus-k8s-1" node="ip-10-0-225-74.us-west-1.compute.internal" evaluatedNodes=6 feasibleNodes=1\nI0514 18:16:02.667269       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-apiserver/apiserver-56f6b658cb-j2m7c" err="0/6 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 3 node(s) didn't match Pod's node affinity."\nI0514 18:16:13.373144       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-oauth-apiserver/apiserver-8559c747c8-hb88q" err="0/6 nodes are available: 1 node(s) were unschedulable, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 3 node(s) didn't match Pod's node affinity."\nE0514 18:16:32.752364       1 leaderelection.go:325] error retrieving resource lock openshift-kube-scheduler/kube-scheduler: etcdserver: request timed out\nI0514 18:16:33.429648       1 scheduler.go:606] "Successfully bound pod to node" pod="openshift-marketplace/community-operators-v4sdq" node="ip-10-0-138-17.us-west-1.compute.internal" evaluatedNodes=6 feasibleNodes=3\nI0514 18:16:33.549506       1 scheduler.go:606] "Successfully bound pod to node" pod="openshift-marketplace/certified-operators-bj2mf" node="ip-10-0-138-17.us-west-1.compute.internal" evaluatedNodes=6 feasibleNodes=3\nI0514 18:16:33.584809       1 leaderelection.go:278] failed to renew lease openshift-kube-scheduler/kube-scheduler: timed out waiting for the condition\nE0514 18:16:33.585252       1 leaderelection.go:301] Failed to release lock: resource name may not be empty\nF0514 18:16:33.585323       1 server.go:217] leaderelection lost\n

May 06 05:32:27.794 E ns/openshift-machine-config-operator pod/machine-config-operator-d67f4bc76-f2fpn node/ip-10-0-157-157.us-east-2.compute.internal container/machine-config-operator container exited with code 255 (Error): eeding\nW0506 05:31:58.413909       1 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding\nW0506 05:31:58.413930       1 reflector.go:436] github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Network ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding\nW0506 05:31:58.413954       1 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.ClusterRole ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding\nW0506 05:31:59.705669       1 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition\nW0506 05:31:59.708199       1 warnings.go:70] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition\nI0506 05:32:26.717490       1 leaderelection.go:278] failed to renew lease openshift-machine-config-operator/machine-config: timed out waiting for the condition\nF0506 05:32:26.717593       1 start.go:102] leaderelection lost\nI0506 05:32:26.721335       1 trace.go:205] Trace[392828266]: "Reflector ListAndWatch" name:github.com/openshift/client-go/config/informers/externalversions/factory.go:101 (06-May-2024 05:31:59.767) (total time: 26953ms):\nTrace[392828266]: [26.953852098s] [26.953852098s] END\nI0506 05:32:26.721413       1 trace.go:205] Trace[1980435746]: "Reflector ListAndWatch" name:github.com/openshift/client-go/config/informers/externalversions/factory.go:101 (06-May-2024 05:31:59.638) (total time: 27082ms):\nTrace[1980435746]: [27.082883276s] [27.082883276s] END\n
May 06 05:34:18.137 E ns/openshift-kube-scheduler pod/openshift-kube-scheduler-ip-10-0-250-57.us-east-2.compute.internal node/ip-10-0-250-57.us-east-2.compute.internal container/kube-scheduler container exited with code 1 (Error): e: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0506 05:33:23.869582       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-etcd/etcd-quorum-guard-56446b67db-bwz8b" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0506 05:33:32.870603       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-apiserver/apiserver-f5cd696b6-498kz" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0506 05:33:32.871131       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-oauth-apiserver/apiserver-59dbdf9669-w7klx" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nI0506 05:33:33.870547       1 factory.go:322] "Unable to schedule pod; no fit; waiting" pod="openshift-etcd/etcd-quorum-guard-56446b67db-bwz8b" err="0/6 nodes are available: 2 node(s) didn't match Pod's node affinity, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 2 node(s) were unschedulable."\nE0506 05:34:15.440657       1 leaderelection.go:325] error retrieving resource lock openshift-kube-scheduler/kube-scheduler: etcdserver: request timed out\nI0506 05:34:17.128803       1 leaderelection.go:278] failed to renew lease openshift-kube-scheduler/kube-scheduler: timed out waiting for the condition\nE0506 05:34:17.128897       1 leaderelection.go:301] Failed to release lock: resource name may not be empty\nF0506 05:34:17.128945       1 server.go:217] leaderelection lost\n
May 06 05:34:28.556 E clusteroperator/monitoring changed Degraded to True: UpdatingGrafanaFailed: Failed to rollout the stack. Error: running task Updating Grafana failed: waiting for Grafana Route to become ready failed: waiting for route openshift-monitoring/grafana: etcdserver: request timed out