May 15 23:07:22.315 E ns/openshift-kni-infra pod/haproxy-host4.cluster11.ocpci.eng.rdu2.redhat.com node/host4.cluster11.ocpci.eng.rdu2.redhat.com uid/8f13d138-e08d-45dd-a4dc-4dac698508c9 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 23:08:43.322 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-15-204102: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2fd9898bd35b33f8624fc6c03fa412d3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-24695d8998f4ff80680fed880d6d5a2c, retrying]
May 15 23:08:43.322 - 139s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-15-204102: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2fd9898bd35b33f8624fc6c03fa412d3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-24695d8998f4ff80680fed880d6d5a2c, retrying]

... 1 lines not shown

May 15 09:18:48.340 E ns/openshift-kni-infra pod/haproxy-host4.cluster16.ocpci.eng.rdu2.redhat.com node/host4.cluster16.ocpci.eng.rdu2.redhat.com uid/dd2950cc-f030-4f03-b64f-8518c5e15099 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 09:20:28.240 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-15-065158: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a774c8126296883fd0770999b4cfd1e7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-631995288c9e636f027099185a5a8e6c, retrying]
May 15 09:20:28.240 - 133s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-15-065158: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a774c8126296883fd0770999b4cfd1e7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-631995288c9e636f027099185a5a8e6c, retrying]

... 1 lines not shown

May 15 00:03:25.782 E ns/openshift-kube-controller-manager pod/kube-controller-manager-host4.cluster3.ocpci.eng.rdu2.redhat.com node/host4.cluster3.ocpci.eng.rdu2.redhat.com uid/ee21349c-5a3f-4517-a74a-819b4a419b5f container/kube-controller-manager reason/ContainerExit code/1 cause/Error b.com/openshift/client-go/image/informers/externalversions/factory.go:101: failed to list *v1.ImageStream: Get "https://api-int.cluster3.ocpci.eng.rdu2.redhat.com:6443/apis/image.openshift.io/v1/imagestreams?resourceVersion=79259": dial tcp: lookup api-int.cluster3.ocpci.eng.rdu2.redhat.com on 10.11.5.160:53: no such host\nE0515 00:03:04.839217       1 reflector.go:140] vendor/github.com/openshift/client-go/image/informers/externalversions/factory.go:101: Failed to watch *v1.ImageStream: failed to list *v1.ImageStream: Get "https://api-int.cluster3.ocpci.eng.rdu2.redhat.com:6443/apis/image.openshift.io/v1/imagestreams?resourceVersion=79259": dial tcp: lookup api-int.cluster3.ocpci.eng.rdu2.redhat.com on 10.11.5.160:53: no such host\nW0515 00:03:04.839214       1 reflector.go:424] vendor/k8s.io/client-go/metadata/metadatainformer/informer.go:90: failed to list *v1.PartialObjectMetadata: Get "https://api-int.cluster3.ocpci.eng.rdu2.redhat.com:6443/apis/machine.openshift.io/v1beta1/machines?resourceVersion=79765": dial tcp: lookup api-int.cluster3.ocpci.eng.rdu2.redhat.com on 10.11.5.160:53: no such host\nE0515 00:03:04.839239       1 reflector.go:140] vendor/k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: Get "https://api-int.cluster3.ocpci.eng.rdu2.redhat.com:6443/apis/machine.openshift.io/v1beta1/machines?resourceVersion=79765": dial tcp: lookup api-int.cluster3.ocpci.eng.rdu2.redhat.com on 10.11.5.160:53: no such host\nE0515 00:03:07.814688       1 leaderelection.go:330] error retrieving resource lock kube-system/kube-controller-manager: Get "https://api-int.cluster3.ocpci.eng.rdu2.redhat.com:6443/api/v1/namespaces/kube-system/configmaps/kube-controller-manager?timeout=5s": context deadline exceeded\nI0515 00:03:07.814753       1 leaderelection.go:283] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition\nE0515 00:03:07.814805       1 controllermanager.go:318] "leaderelection lost"\n
May 15 00:05:27.260 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-14-213601: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ef0f64e70ea0f1a3b48c6b196e858690 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ba7ab6359ca1daa5c85bb01d38227986, retrying]
May 15 00:05:27.260 - 120s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-14-213601: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ef0f64e70ea0f1a3b48c6b196e858690 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ba7ab6359ca1daa5c85bb01d38227986, retrying]

... 1 lines not shown

May 14 15:48:04.296 E ns/openshift-kni-infra pod/haproxy-host4.cluster9.ocpci.eng.rdu2.redhat.com node/host4.cluster9.ocpci.eng.rdu2.redhat.com uid/639fdcde-a9fb-46e1-8cc6-0ea30ae63dfc container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 15:49:32.486 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-14-132144: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b1234323d6a9730708a25ea89404edc5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3ab92b8e87a39b44fa6798c9af4347c3, retrying]
May 14 15:49:32.486 - 126s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-14-132144: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b1234323d6a9730708a25ea89404edc5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3ab92b8e87a39b44fa6798c9af4347c3, retrying]

... 1 lines not shown

May 10 15:01:33.737 E ns/openshift-multus pod/multus-additional-cni-plugins-4sw2v node/host5.cluster3.ocpci.eng.rdu2.redhat.com uid/0ac93663-42a7-4edf-aa55-de06d7790370 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 15:01:42.035 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-10-124007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e49648014d75d10ec02e4521b3852e2c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-47648ca79283a345a259c9b8b5f653c1, retrying]
May 10 15:01:42.035 - 491s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-10-124007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e49648014d75d10ec02e4521b3852e2c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-47648ca79283a345a259c9b8b5f653c1, retrying]

... 1 lines not shown

May 11 04:54:21.088 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-84lf8 node/host5.cluster15.ocpci.eng.rdu2.redhat.com uid/ec90da8d-4fc8-4f74-a89a-56e37238edee container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 11 04:55:20.150 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-11-022650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8fa41caf49113595e5cb2b5f8ebaf8a1 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-241ea4f10111c6fc216a5a9277b5b91e, retrying]
May 11 04:55:20.150 - 153s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-11-022650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8fa41caf49113595e5cb2b5f8ebaf8a1 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-241ea4f10111c6fc216a5a9277b5b91e, retrying]

... 1 lines not shown

May 10 07:50:00.531 E ns/openshift-kni-infra pod/haproxy-host4.cluster11.ocpci.eng.rdu2.redhat.com node/host4.cluster11.ocpci.eng.rdu2.redhat.com uid/8c08a40e-c127-498f-91cf-2f6345bd5d9f container/verify-api-int-resolvable reason/ContainerExit code/1 cause/Error
May 10 07:51:30.898 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-10-052846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a90dc1fdd62adefc4ff45c765bdd8a02 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-7be2627fb4210fd01ec857ba96cc18fd, retrying]
May 10 07:51:30.898 - 129s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-10-052846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a90dc1fdd62adefc4ff45c765bdd8a02 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-7be2627fb4210fd01ec857ba96cc18fd, retrying]

... 1 lines not shown

May 10 01:51:06.259 E ns/openshift-machine-api pod/metal3-5865864dd7-5z9cn node/host4.cluster6.ocpci.eng.rdu2.redhat.com uid/193ecceb-3fe7-4d0b-aa1a-b137f7dc7cba container/metal3-static-ip-set reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 01:51:10.027 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-09-232846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1ed065123a16b381fad2c6f76ae64245 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-330433264f580515140b57c33fd93b13, retrying]
May 10 01:51:10.027 - 628s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-09-232846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1ed065123a16b381fad2c6f76ae64245 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-330433264f580515140b57c33fd93b13, retrying]

... 1 lines not shown

May 09 14:27:16.277 E ns/openshift-kni-infra pod/haproxy-host2.cluster9.ocpci.eng.rdu2.redhat.com node/host2.cluster9.ocpci.eng.rdu2.redhat.com uid/d8dfbea5-2baa-420a-b6d6-949d880aa52b container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 14:28:25.749 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-61aac0f9af4570d2a0af910f30b7e5a8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3b97686f2482e700ec81783ba5a3e45d, retrying]
May 09 14:28:25.749 - 521s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-61aac0f9af4570d2a0af910f30b7e5a8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3b97686f2482e700ec81783ba5a3e45d, retrying]

... 1 lines not shown

May 08 15:03:11.879 E ns/openshift-kni-infra pod/haproxy-host4.cluster11.ocpci.eng.rdu2.redhat.com node/host4.cluster11.ocpci.eng.rdu2.redhat.com uid/5fc155e6-ce04-4190-ad1e-e64476046fcd container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 15:04:39.696 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-08-124615: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8ac26e23462a793dab73a04f3d6a8b03 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6e882647b6b20d565d8343c3fe05be01, retrying]
May 08 15:04:39.696 - 529s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-08-124615: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8ac26e23462a793dab73a04f3d6a8b03 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6e882647b6b20d565d8343c3fe05be01, retrying]

... 1 lines not shown

May 08 05:29:38.385 E ns/openshift-console pod/console-95476cc58-2b4kk node/host2.cluster7.ocpci.eng.rdu2.redhat.com uid/6d4054c6-d333-454f-9f33-2f81a7b5609b container/console reason/ContainerExit code/2 cause/Error W0508 05:09:19.655552       1 main.go:228] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!\nI0508 05:09:19.655572       1 main.go:348] cookies are secure!\nI0508 05:09:19.677020       1 main.go:809] Binding to [::]:8443...\nI0508 05:09:19.677044       1 main.go:811] using TLS\n
May 08 05:33:11.994 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-08-030344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9c89815a47e292747b49ad21417c6dab expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-619c84062dfa70fef68728498740cbad, retrying]
May 08 05:33:11.994 - 125s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-08-030344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9c89815a47e292747b49ad21417c6dab expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-619c84062dfa70fef68728498740cbad, retrying]

... 1 lines not shown