May 16 21:40:19.190 - 7s    E clusteroperator/network condition/Degraded status/True reason/Error while updating operator configuration: could not apply (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /whereabouts-cni: failed to apply / update (rbac.authorization.k8s.io/v1, Kind=ClusterRole) /whereabouts-cni: Patch "https://api-int.cluster7.ocpci.eng.rdu2.redhat.com:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/whereabouts-cni?fieldManager=cluster-network-operator%2Foperconfig&force=true": write tcp 10.10.129.61:54646->10.10.129.64:6443: write: connection reset by peer
May 16 21:41:59.155 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-191307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eecbc20d272715403bad5185b05fcb1a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-99b386c6767cf007b76bab34f5e3ac85, retrying]
May 16 21:41:59.155 - 118s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-191307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eecbc20d272715403bad5185b05fcb1a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-99b386c6767cf007b76bab34f5e3ac85, retrying]

... 1 lines not shown

May 16 07:49:12.203 E ns/openshift-kni-infra pod/haproxy-host3.cluster13.ocpci.eng.rdu2.redhat.com node/host3.cluster13.ocpci.eng.rdu2.redhat.com uid/39ad4364-da06-4979-9aa0-4adc71fa941b container/verify-api-int-resolvable reason/ContainerExit code/6 cause/Error
May 16 07:51:12.130 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-052445: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7238aa3af72726515a2ecc76f4448a89 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-537be40883f3af7c439bb9ea7940225f, retrying]
May 16 07:51:12.130 - 118s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-052445: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7238aa3af72726515a2ecc76f4448a89 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-537be40883f3af7c439bb9ea7940225f, retrying]

... 1 lines not shown

May 16 15:57:41.732 E ns/openshift-machine-api pod/metal3-79fc56f696-r7sws node/host4.cluster14.ocpci.eng.rdu2.redhat.com uid/fd86467a-179c-41e0-9bee-e0b070103267 container/metal3-static-ip-set reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 15:59:30.857 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-131307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ab6098c98cbc18dde0af8d189e581ab2 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8aba40634a74da4149c7fd6ced38b444, retrying]
May 16 15:59:30.857 - 106s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-131307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ab6098c98cbc18dde0af8d189e581ab2 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8aba40634a74da4149c7fd6ced38b444, retrying]

... 1 lines not shown

May 15 19:15:12.194 E ns/openshift-machine-api pod/metal3-575f6db498-6mkvv node/host2.cluster16.ocpci.eng.rdu2.redhat.com uid/9c377c0d-2408-4dee-8965-2fb96f1f9f62 container/metal3-static-ip-set reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 19:16:05.702 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-15-165412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c7ab127bda44fdc064c04fb7e83b5987 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-07f29fec34c3d62d5ec4a61be7c00e90, retrying]
May 15 19:16:05.702 - 575s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-15-165412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c7ab127bda44fdc064c04fb7e83b5987 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-07f29fec34c3d62d5ec4a61be7c00e90, retrying]

... 1 lines not shown

May 15 13:29:44.293 E ns/openshift-machine-api pod/metal3-56b877cd44-fjgwn node/host4.cluster6.ocpci.eng.rdu2.redhat.com uid/990e7d50-377c-4c38-990a-b72e9323d77d container/metal3-static-ip-set reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 13:31:43.849 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-15-105412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b89546a1f1971d09f9c71f870904bbc4 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2cf76f26e456e05e4e1eb02df93c0abd, retrying]
May 15 13:31:43.849 - 109s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-15-105412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b89546a1f1971d09f9c71f870904bbc4 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2cf76f26e456e05e4e1eb02df93c0abd, retrying]

... 1 lines not shown

May 10 16:01:39.971 E ns/openshift-kube-controller-manager pod/kube-controller-manager-host3.cluster9.ocpci.eng.rdu2.redhat.com node/host3.cluster9.ocpci.eng.rdu2.redhat.com uid/7b8112e3-c9e1-4ef8-94e1-9b3602cacb1e container/kube-controller-manager reason/ContainerExit code/1 cause/Error p api-int.cluster9.ocpci.eng.rdu2.redhat.com on 10.11.5.160:53: no such host\nE0510 16:00:49.890419       1 reflector.go:140] vendor/k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: Get "https://api-int.cluster9.ocpci.eng.rdu2.redhat.com:6443/apis/infrastructure.cluster.x-k8s.io/v1beta1/metal3remediationtemplates?resourceVersion=73234": dial tcp: lookup api-int.cluster9.ocpci.eng.rdu2.redhat.com on 10.11.5.160:53: no such host\nW0510 16:00:49.904835       1 reflector.go:424] vendor/k8s.io/client-go/metadata/metadatainformer/informer.go:90: failed to list *v1.PartialObjectMetadata: Get "https://api-int.cluster9.ocpci.eng.rdu2.redhat.com:6443/apis/operators.coreos.com/v1/operatorgroups?resourceVersion=73183": dial tcp: lookup api-int.cluster9.ocpci.eng.rdu2.redhat.com on 10.11.5.160:53: no such host\nE0510 16:00:49.904867       1 reflector.go:140] vendor/k8s.io/client-go/metadata/metadatainformer/informer.go:90: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: Get "https://api-int.cluster9.ocpci.eng.rdu2.redhat.com:6443/apis/operators.coreos.com/v1/operatorgroups?resourceVersion=73183": dial tcp: lookup api-int.cluster9.ocpci.eng.rdu2.redhat.com on 10.11.5.160:53: no such host\nE0510 16:00:54.389411       1 leaderelection.go:330] error retrieving resource lock kube-system/kube-controller-manager: Get "https://api-int.cluster9.ocpci.eng.rdu2.redhat.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=6s": context deadline exceeded\nI0510 16:00:54.389448       1 leaderelection.go:283] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition\nE0510 16:00:54.389499       1 controllermanager.go:328] "leaderelection lost"\nI0510 16:00:54.389506       1 pv_controller_base.go:334] Shutting down persistent volume controller\nI0510 16:00:54.389520       1 replica_set.go:213] Shutting down replicationcontroller controller\n
May 10 16:02:59.110 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-10-134150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dc0efb14ba660e928e9ce934e8b65224 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a62d1caaa46f889a2e5da2a341ac92e8, retrying]
May 10 16:02:59.110 - 523s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-10-134150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dc0efb14ba660e928e9ce934e8b65224 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a62d1caaa46f889a2e5da2a341ac92e8, retrying]

... 1 lines not shown

May 14 10:27:22.552 E ns/openshift-monitoring pod/alertmanager-main-0 node/host5.cluster6.ocpci.eng.rdu2.redhat.com uid/6b141984-dd31-4126-8874-4f09528f371f container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-14T10:27:20.054Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=38194c1bde7b693bb84526d1084400a1e76da31b)"\nts=2024-05-14T10:27:20.054Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@cbb9d02780b4, date=20240510-00:07:28)"\nts=2024-05-14T10:27:20.072Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-14T10:27:20.120Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-14T10:27:20.120Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-14T10:27:20.120Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=48.082619ms\n
May 14 10:27:32.355 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-14-080341: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-35cc69f46e9bdc55e835fb13e3b4d320 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f337d7eb229b5ee27c3eb5b4a19ea275, retrying]
May 14 10:27:32.355 - 508s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-14-080341: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-35cc69f46e9bdc55e835fb13e3b4d320 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f337d7eb229b5ee27c3eb5b4a19ea275, retrying]

... 1 lines not shown

May 14 16:36:02.188 E ns/openshift-kni-infra pod/haproxy-host3.cluster6.ocpci.eng.rdu2.redhat.com node/host3.cluster6.ocpci.eng.rdu2.redhat.com uid/935f1061-267a-452c-8ffa-c66d06954c1d container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 16:37:39.564 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-14-140838: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b18bf9b0fd46cfe5a855a6db271574ec expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2c60ee30393352b0c4681b82cba95e0b, retrying]
May 14 16:37:39.564 - 138s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-14-140838: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b18bf9b0fd46cfe5a855a6db271574ec expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2c60ee30393352b0c4681b82cba95e0b, retrying]

... 1 lines not shown

May 09 15:44:22.547 - 65s   E clusteroperator/dns condition/Degraded status/True reason/DNS default is degraded
May 09 15:46:16.156 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-09-130302: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a9eb85553d2d502761f7290fff658711 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-347f0ac64f79c7e3fe04468afa449339, retrying]
May 09 15:46:16.156 - 118s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-09-130302: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a9eb85553d2d502761f7290fff658711 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-347f0ac64f79c7e3fe04468afa449339, retrying]

... 1 lines not shown

May 08 18:01:19.582 E ns/openshift-monitoring pod/alertmanager-main-1 node/host5.cluster5.ocpci.eng.rdu2.redhat.com uid/583b8bcc-d61d-4050-b52a-1376e53d1eb9 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T18:01:17.381Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=c5f095c646ea4c1f9524833e8e3579b827a501aa)"\nts=2024-05-08T18:01:17.381Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@929f9e3118d2, date=20240425-19:10:00)"\nts=2024-05-08T18:01:17.389Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T18:01:17.410Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T18:01:17.410Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T18:01:17.410Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=20.954841ms\n
May 08 18:01:27.868 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-153833: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-399fd1bf0c340c4dac609d4e88846ffa expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d523e55d2567ff874fa26a5e4c1f0778, retrying]
May 08 18:01:27.868 - 490s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-153833: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-399fd1bf0c340c4dac609d4e88846ffa expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d523e55d2567ff874fa26a5e4c1f0778, retrying]

... 1 lines not shown

May 10 07:20:36.038 E ns/openshift-machine-api pod/metal3-76699c9d84-k9lr9 node/host3.cluster8.ocpci.eng.rdu2.redhat.com uid/3afa63a3-de90-47d0-a3d8-ca8017708c55 container/metal3-static-ip-set reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 07:21:59.937 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-10-045757: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6986b917dde23605b6be9653b9cdd9d3 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3c2a0d5bf78ed1eb788a54b497c029b7, retrying]
May 10 07:21:59.937 - 507s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-10-045757: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6986b917dde23605b6be9653b9cdd9d3 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3c2a0d5bf78ed1eb788a54b497c029b7, retrying]

... 1 lines not shown

May 08 11:46:36.465 E ns/openshift-kni-infra pod/haproxy-host2.cluster9.ocpci.eng.rdu2.redhat.com node/host2.cluster9.ocpci.eng.rdu2.redhat.com uid/dda798ed-2dfd-488e-9cec-d619565182dd container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 11:48:37.459 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-090908: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-de3b121f658b71f257dce01f15b861a9 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-7d49cf34da31474117a9bd936cef99d7, retrying]
May 08 11:48:37.459 - 116s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-090908: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-de3b121f658b71f257dce01f15b861a9 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-7d49cf34da31474117a9bd936cef99d7, retrying]

... 1 lines not shown

May 08 05:20:17.085 E ns/openshift-machine-api pod/metal3-864f758fd6-lbd2l node/host4.cluster8.ocpci.eng.rdu2.redhat.com uid/6f727391-e97e-4e29-a806-62a4115d4c3e container/metal3-static-ip-set reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 05:22:12.431 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-025210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0bca088df9524f675a136f6ef2a19767 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-37e7d2d06843b34ccb08bf32ca534c84, retrying]
May 08 05:22:12.431 - 118s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-025210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0bca088df9524f675a136f6ef2a19767 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-37e7d2d06843b34ccb08bf32ca534c84, retrying]

... 1 lines not shown