May 18 13:04:12.052 E ns/openshift-ovn-kubernetes pod/ovnkube-master-nvkdh node/ip-10-0-135-222.us-west-2.compute.internal uid/d053d7d6-c07b-4f91-a196-27bf6d07dd7e container/northd reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 18 13:04:14.488 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-7c9530c70f27658b3234d9f80830180b expected: registry.ci.openshift.org/ocp/4.12-2024-05-16-010831@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150, retrying]
May 18 13:04:14.488 - 338s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-7c9530c70f27658b3234d9f80830180b expected: registry.ci.openshift.org/ocp/4.12-2024-05-16-010831@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150, retrying]

... 1 lines not shown

May 16 12:52:12.295 - 49s   E clusteroperator/etcd condition/Degraded status/True reason/ClusterMemberControllerDegraded: unhealthy members found during reconciling members\nEtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:3880919596540586521 name:"ip-10-0-209-160.us-west-1.compute.internal" peerURLs:"https://10.0.209.160:2380" clientURLs:"https://10.0.209.160:2379"  Healthy:true Took:2.764409ms Error:<nil>} {Member:ID:7466023461321759843 name:"ip-10-0-140-58.us-west-1.compute.internal" peerURLs:"https://10.0.140.58:2380" clientURLs:"https://10.0.140.58:2379"  Healthy:true Took:2.488155ms Error:<nil>} {Member:ID:14661298595357366462 name:"ip-10-0-186-231.us-west-1.compute.internal" peerURLs:"https://10.0.186.231:2380" clientURLs:"https://10.0.186.231:2379"  Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.186.231:2379]: context deadline exceeded}]\nEtcdMembersDegraded: 2 of 3 members are available, ip-10-0-186-231.us-west-1.compute.internal is unhealthy\nNodeControllerDegraded: The master nodes not ready: node "ip-10-0-186-231.us-west-1.compute.internal" not ready since 2024-05-16 12:50:54 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
May 16 12:52:25.398 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-eb62ff81e3cf2ca6a7e02064c7d0821a expected: registry.ci.openshift.org/ocp/4.12-2024-05-16-010831@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150, retrying]
May 16 12:52:25.398 - 389s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-eb62ff81e3cf2ca6a7e02064c7d0821a expected: registry.ci.openshift.org/ocp/4.12-2024-05-16-010831@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150, retrying]

... 1 lines not shown

May 17 12:56:56.213 E ns/e2e-k8s-sig-apps-daemonset-upgrade-1595 pod/ds1-8xgz9 node/ip-10-0-135-41.us-west-1.compute.internal uid/42c6874e-ed11-433c-a80b-0439bdd4c2fb container/app reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 17 12:56:56.474 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-2645e13e77986a03f6ff224bde813879 expected: registry.ci.openshift.org/ocp/4.12-2024-05-16-010831@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150, retrying]
May 17 12:56:56.474 - 332s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-2645e13e77986a03f6ff224bde813879 expected: registry.ci.openshift.org/ocp/4.12-2024-05-16-010831@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150, retrying]

... 1 lines not shown

May 16 02:50:30.381 E ns/openshift-e2e-loki pod/loki-promtail-j8kq7 node/ip-10-0-154-237.us-west-1.compute.internal uid/5ba0521c-c0da-4c50-b357-61a4d295616c container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 02:50:48.733 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-e2b0cd4536b9098c3ab70d6aeb4e258f expected: registry.ci.openshift.org/ocp/4.12-2024-05-16-010831@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150, retrying]
May 16 02:50:48.733 - 303s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-e2b0cd4536b9098c3ab70d6aeb4e258f expected: registry.ci.openshift.org/ocp/4.12-2024-05-16-010831@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150, retrying]

... 1 lines not shown

May 15 11:23:43.524 E ns/openshift-e2e-loki pod/loki-promtail-hgwjw node/ip-10-0-202-125.us-west-1.compute.internal uid/0e9eec61-fe1e-4362-9bee-172db4fdca98 container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 11:24:11.945 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-15-094441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-e77d7aa6b117b716ebf5f5381e9d1ec4 expected: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-14-122945@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 15 11:24:11.945 - 372s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-15-094441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-e77d7aa6b117b716ebf5f5381e9d1ec4 expected: registry.ci.openshift.org/ocp/4.12-2024-05-15-094441@sha256:2fdebef3f6fb8fb7a79931485ee7d267e84edf309f48dd5b7f3d15e46cd98150 got: registry.ci.openshift.org/ocp/4.12-2024-05-14-122945@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 13 23:33:53.080 E ns/openshift-cluster-storage-operator pod/csi-snapshot-controller-ccc7c44f-8wbm5 node/ip-10-0-247-212.ec2.internal uid/9348856d-0a01-4a40-b021-dea7cb7d8837 container/snapshot-controller reason/ContainerExit code/2 cause/Error
May 13 23:33:56.154 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-31a55d60af958df0b6b921956a8d21b5 expected: registry.ci.openshift.org/ocp/4.12-2024-05-13-215426@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-13-090953@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 13 23:33:56.154 - 297s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-31a55d60af958df0b6b921956a8d21b5 expected: registry.ci.openshift.org/ocp/4.12-2024-05-13-215426@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-13-090953@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 13 10:50:38.697 - 37s   E clusteroperator/etcd condition/Degraded status/True reason/ClusterMemberControllerDegraded: unhealthy members found during reconciling members\nEtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:2347489532482104167 name:"ip-10-0-144-197.us-west-1.compute.internal" peerURLs:"https://10.0.144.197:2380" clientURLs:"https://10.0.144.197:2379"  Healthy:true Took:2.331743ms Error:<nil>} {Member:ID:4737620624948772896 name:"ip-10-0-153-237.us-west-1.compute.internal" peerURLs:"https://10.0.153.237:2380" clientURLs:"https://10.0.153.237:2379"  Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.153.237:2379]: context deadline exceeded} {Member:ID:7421708256290558271 name:"ip-10-0-209-205.us-west-1.compute.internal" peerURLs:"https://10.0.209.205:2380" clientURLs:"https://10.0.209.205:2379"  Healthy:true Took:2.50291ms Error:<nil>}]\nEtcdMembersDegraded: 2 of 3 members are available, ip-10-0-153-237.us-west-1.compute.internal is unhealthy\nNodeControllerDegraded: The master nodes not ready: node "ip-10-0-153-237.us-west-1.compute.internal" not ready since 2024-05-13 10:49:08 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
May 13 10:50:38.755 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-090953: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-d9695d0468ae25a8098c7ad966f3454a expected: registry.ci.openshift.org/ocp/4.12-2024-05-13-090953@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-11-181333@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 13 10:50:38.755 - 372s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-090953: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-d9695d0468ae25a8098c7ad966f3454a expected: registry.ci.openshift.org/ocp/4.12-2024-05-13-090953@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-11-181333@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 14 13:01:49.039 E ns/openshift-e2e-loki pod/loki-promtail-njcqp node/ip-10-0-168-100.us-west-1.compute.internal uid/2e783383-1fa0-4afc-b35e-6e53dfc00374 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 13:01:55.225 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-7590898f28ff99df26d760894d2c596e expected: registry.ci.openshift.org/ocp/4.12-2024-05-13-215426@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-13-090953@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 14 13:01:55.225 - 331s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-7590898f28ff99df26d760894d2c596e expected: registry.ci.openshift.org/ocp/4.12-2024-05-13-215426@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-13-090953@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 14 14:22:06.930 E ns/openshift-ovn-kubernetes pod/ovnkube-master-nl7qn node/ip-10-0-222-65.us-west-2.compute.internal uid/d7fb2201-548a-4bd8-8629-650f337fae25 container/northd reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 14:22:09.804 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-273a5397c61c9d731a5ac2eff47dd82f expected: registry.ci.openshift.org/ocp/4.12-2024-05-14-122945@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-13-215426@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 14 14:22:09.804 - 21s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-273a5397c61c9d731a5ac2eff47dd82f expected: registry.ci.openshift.org/ocp/4.12-2024-05-14-122945@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-13-215426@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 11 19:52:45.598 - 34s   E clusteroperator/etcd condition/Degraded status/True reason/ClusterMemberControllerDegraded: unhealthy members found during reconciling members\nEtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:1874041976332156421 name:"ip-10-0-146-188.us-east-2.compute.internal" peerURLs:"https://10.0.146.188:2380" clientURLs:"https://10.0.146.188:2379"  Healthy:true Took:1.343366ms Error:<nil>} {Member:ID:14816436146890546105 name:"ip-10-0-173-198.us-east-2.compute.internal" peerURLs:"https://10.0.173.198:2380" clientURLs:"https://10.0.173.198:2379"  Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.173.198:2379]: context deadline exceeded} {Member:ID:16153318737982119043 name:"ip-10-0-198-48.us-east-2.compute.internal" peerURLs:"https://10.0.198.48:2380" clientURLs:"https://10.0.198.48:2379"  Healthy:true Took:1.111444ms Error:<nil>}]\nEtcdMembersDegraded: 2 of 3 members are available, ip-10-0-173-198.us-east-2.compute.internal is unhealthy\nNodeControllerDegraded: The master nodes not ready: node "ip-10-0-173-198.us-east-2.compute.internal" not ready since 2024-05-11 19:51:04 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
May 11 19:52:57.631 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-f5af7dacdc7612c7f4efdf9c9edd0d61 expected: registry.ci.openshift.org/ocp/4.12-2024-05-11-181333@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-11-061012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 11 19:52:57.631 - 340s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-f5af7dacdc7612c7f4efdf9c9edd0d61 expected: registry.ci.openshift.org/ocp/4.12-2024-05-11-181333@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-11-061012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 11 13:01:32.394 E ns/openshift-e2e-loki pod/loki-promtail-x5d8t node/ip-10-0-156-240.ec2.internal uid/b1b6befb-e80c-48bd-aa60-6d253dde95d1 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 11 13:01:33.425 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-e4c5326819dcd173abf6a3393bc4904b expected: registry.ci.openshift.org/ocp/4.12-2024-05-11-001012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-10-132619@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 11 13:01:33.425 - 15s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-e4c5326819dcd173abf6a3393bc4904b expected: registry.ci.openshift.org/ocp/4.12-2024-05-11-001012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-10-132619@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 11 02:11:49.649 E ns/openshift-kube-apiserver pod/kube-apiserver-ip-10-0-207-88.us-west-2.compute.internal node/ip-10-0-207-88.us-west-2.compute.internal uid/efc6ab29-5d71-4531-b61f-148d36140de1 container/kube-apiserver-check-endpoints reason/ContainerExit code/255 cause/Error W0511 02:11:47.345267       1 cmd.go:216] Using insecure, self-signed certificates\nI0511 02:11:47.354290       1 crypto.go:601] Generating new CA for check-endpoints-signer@1715393507 cert, and key in /tmp/serving-cert-2427026460/serving-signer.crt, /tmp/serving-cert-2427026460/serving-signer.key\nI0511 02:11:47.933064       1 observer_polling.go:159] Starting file observer\nW0511 02:11:47.942358       1 builder.go:239] unable to get owner reference (falling back to namespace): Get "https://localhost:6443/api/v1/namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-207-88.us-west-2.compute.internal": dial tcp [::1]:6443: connect: connection refused\nI0511 02:11:47.942533       1 builder.go:271] check-endpoints version v4.0.0-alpha.0-1741-g09d7ddb-09d7ddbab\nI0511 02:11:47.950002       1 dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-cert::/tmp/serving-cert-2427026460/tls.crt::/tmp/serving-cert-2427026460/tls.key"\nF0511 02:11:48.207126       1 cmd.go:141] error initializing delegating authentication: unable to load configmap based request-header-client-ca-file: Get "https://localhost:6443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp [::1]:6443: connect: connection refused\n
May 11 02:11:52.158 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-c8090c40308a82f970aca4adb4e15670 expected: registry.ci.openshift.org/ocp/4.12-2024-05-11-001012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-10-132619@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 11 02:11:52.158 - 23s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-c8090c40308a82f970aca4adb4e15670 expected: registry.ci.openshift.org/ocp/4.12-2024-05-11-001012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-10-132619@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 11 08:02:36.590 E ns/openshift-machine-config-operator pod/machine-config-daemon-bnm7f node/ip-10-0-168-121.us-west-1.compute.internal uid/915190db-ccad-4614-80e1-ea4f37770b27 container/machine-config-daemon reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 11 08:02:36.764 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-49c34a6c608b66df248870aff2d822c0 expected: registry.ci.openshift.org/ocp/4.12-2024-05-11-061012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-11-001012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]
May 11 08:02:36.764 - 357s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-49c34a6c608b66df248870aff2d822c0 expected: registry.ci.openshift.org/ocp/4.12-2024-05-11-061012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-11-001012@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730, retrying]

... 1 lines not shown

May 10 06:29:13.803 E ns/openshift-cluster-csi-drivers pod/aws-ebs-csi-driver-node-69mkk node/ip-10-0-208-255.us-west-2.compute.internal uid/41bd59bb-52b8-411b-a8fc-7d2e4ca36fb9 container/csi-node-driver-registrar reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 06:29:15.258 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-0a61e7c3817e50be3a4dbeb7ab740b0e expected: registry.ci.openshift.org/ocp/4.12-2024-05-10-043105@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-09-061655@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3, retrying]
May 10 06:29:15.258 - 523s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-0a61e7c3817e50be3a4dbeb7ab740b0e expected: registry.ci.openshift.org/ocp/4.12-2024-05-10-043105@sha256:c11a201f7b966db3361abcf86807fbd4bfaad290839119a26548d9abcb430730 got: registry.ci.openshift.org/ocp/4.12-2024-05-09-061655@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3, retrying]

... 1 lines not shown

May 09 17:01:28.578 E ns/openshift-multus pod/multus-additional-cni-plugins-n784z node/ip-10-0-241-100.us-west-2.compute.internal uid/2e7c3c3d-fa9e-4958-93e2-5b0c9b58e0d8 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 17:01:41.045 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-4f093e808ca7271ab8636405b6d87a1d expected: registry.ci.openshift.org/ocp/4.12-2024-05-09-151252@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-09-061655@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3, retrying]
May 09 17:01:41.045 - 7s    E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-4f093e808ca7271ab8636405b6d87a1d expected: registry.ci.openshift.org/ocp/4.12-2024-05-09-151252@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-09-061655@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3, retrying]

... 1 lines not shown

May 09 13:02:48.809 - 30s   E clusteroperator/etcd condition/Degraded status/True reason/ClusterMemberControllerDegraded: unhealthy members found during reconciling members\nEtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:8145318818150958050 name:"ip-10-0-221-185.us-east-2.compute.internal" peerURLs:"https://10.0.221.185:2380" clientURLs:"https://10.0.221.185:2379"  Healthy:true Took:2.390546ms Error:<nil>} {Member:ID:13496203047329341821 name:"ip-10-0-159-243.us-east-2.compute.internal" peerURLs:"https://10.0.159.243:2380" clientURLs:"https://10.0.159.243:2379"  Healthy:true Took:1.307851ms Error:<nil>} {Member:ID:15950040909993789541 name:"ip-10-0-137-27.us-east-2.compute.internal" peerURLs:"https://10.0.137.27:2380" clientURLs:"https://10.0.137.27:2379"  Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.137.27:2379]: context deadline exceeded}]\nEtcdMembersDegraded: 2 of 3 members are available, ip-10-0-137-27.us-east-2.compute.internal is unhealthy\nNodeControllerDegraded: The master nodes not ready: node "ip-10-0-137-27.us-east-2.compute.internal" not ready since 2024-05-09 13:01:17 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
May 09 13:02:54.840 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-a6146604726b4cd4f91f54028aa9e24e expected: registry.ci.openshift.org/ocp/4.12-2024-05-08-124617@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-08-023912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]
May 09 13:02:54.840 - 40s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-a6146604726b4cd4f91f54028aa9e24e expected: registry.ci.openshift.org/ocp/4.12-2024-05-08-124617@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-08-023912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]

... 1 lines not shown

May 09 08:16:49.227 E ns/openshift-network-diagnostics pod/network-check-target-f9prx node/ip-10-0-175-222.us-east-2.compute.internal uid/6b75dd08-d2da-42f0-ad8a-71bbdc3b9daf container/network-check-target-container reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 08:16:50.661 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-061655: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-7dc7637e21c997068846849eca62ad07 expected: registry.ci.openshift.org/ocp/4.12-2024-05-09-061655@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-08-124617@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3, retrying]
May 09 08:16:50.661 - 19s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-061655: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-7dc7637e21c997068846849eca62ad07 expected: registry.ci.openshift.org/ocp/4.12-2024-05-09-061655@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-08-124617@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3, retrying]

... 1 lines not shown

May 08 21:31:56.627 E ns/openshift-dns pod/dns-default-bc5jw node/ip-10-0-158-39.us-east-2.compute.internal uid/18326deb-856f-48e9-b9ce-1b538a352abf container/dns reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 21:32:00.430 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-e80f5f4fd2370ae62f628d9ae104babb expected: registry.ci.openshift.org/ocp/4.12-2024-05-08-194829@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-08-124617@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3, retrying]
May 08 21:32:00.430 - 320s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-e80f5f4fd2370ae62f628d9ae104babb expected: registry.ci.openshift.org/ocp/4.12-2024-05-08-194829@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-08-124617@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3, retrying]

... 1 lines not shown

May 08 14:46:08.570 - 34s   E clusteroperator/etcd condition/Degraded status/True reason/ClusterMemberControllerDegraded: unhealthy members found during reconciling members\nEtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:5669710001800676757 name:"ip-10-0-188-110.us-west-2.compute.internal" peerURLs:"https://10.0.188.110:2380" clientURLs:"https://10.0.188.110:2379"  Healthy:true Took:1.212875ms Error:<nil>} {Member:ID:12116063025859780360 name:"ip-10-0-201-29.us-west-2.compute.internal" peerURLs:"https://10.0.201.29:2380" clientURLs:"https://10.0.201.29:2379"  Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.201.29:2379]: context deadline exceeded} {Member:ID:13978493049161650815 name:"ip-10-0-139-150.us-west-2.compute.internal" peerURLs:"https://10.0.139.150:2380" clientURLs:"https://10.0.139.150:2379"  Healthy:true Took:4.323086ms Error:<nil>}]\nEtcdMembersDegraded: 2 of 3 members are available, ip-10-0-201-29.us-west-2.compute.internal is unhealthy\nNodeControllerDegraded: The master nodes not ready: node "ip-10-0-201-29.us-west-2.compute.internal" not ready since 2024-05-08 14:44:38 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
May 08 14:46:09.621 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-60af2abc6c78520f753e41816c1c5bc2 expected: registry.ci.openshift.org/ocp/4.12-2024-05-08-124617@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-08-023912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]
May 08 14:46:09.621 - 47s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-60af2abc6c78520f753e41816c1c5bc2 expected: registry.ci.openshift.org/ocp/4.12-2024-05-08-124617@sha256:3bdf7778d44e52167d1d959048c6831e4d151d61a2240154161dd7fa0ddee9c3 got: registry.ci.openshift.org/ocp/4.12-2024-05-08-023912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]

... 1 lines not shown

May 08 13:08:55.431 E ns/openshift-ovn-kubernetes pod/ovnkube-master-cdhss node/ip-10-0-163-253.us-east-2.compute.internal uid/2d8d5dbb-3a9b-4502-ad27-d91e4c4d378c container/ovnkube-master reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 13:09:03.003 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-07-203912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-55173103c8c7f4eb1b98e6b4dfbc246c expected: registry.ci.openshift.org/ocp/4.12-2024-05-07-203912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-07-143912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]
May 08 13:09:03.003 - 19s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-07-203912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-55173103c8c7f4eb1b98e6b4dfbc246c expected: registry.ci.openshift.org/ocp/4.12-2024-05-07-203912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-07-143912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]

... 1 lines not shown

May 07 16:34:45.046 E ns/openshift-kube-controller-manager pod/kube-controller-manager-ip-10-0-191-131.us-west-1.compute.internal node/ip-10-0-191-131.us-west-1.compute.internal uid/d33f09da-f0e7-48de-98e4-f895eebd211f container/kube-controller-manager reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 07 16:34:50.526 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-07-143912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-ce000738fb4a1d83b40a288ddc346a70 expected: registry.ci.openshift.org/ocp/4.12-2024-05-07-143912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-06-183542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]
May 07 16:34:50.526 - 16s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-07-143912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-ce000738fb4a1d83b40a288ddc346a70 expected: registry.ci.openshift.org/ocp/4.12-2024-05-07-143912@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-06-183542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]

... 1 lines not shown

May 07 12:53:34.863 E ns/openshift-ovn-kubernetes pod/ovnkube-master-j9qnp node/ip-10-0-156-43.us-east-2.compute.internal uid/efabdc6b-8f86-4b9a-9355-73e4550341a8 container/nbdb reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 07 12:53:39.361 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-29b60fd57bf82133ec630be02d6994c9 expected: registry.ci.openshift.org/ocp/4.12-2024-05-06-183542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-06-123542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]
May 07 12:53:39.361 - 378s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-29b60fd57bf82133ec630be02d6994c9 expected: registry.ci.openshift.org/ocp/4.12-2024-05-06-183542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-06-123542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]

... 1 lines not shown

May 06 14:15:38.141 E ns/openshift-e2e-loki pod/loki-promtail-8lcks node/ip-10-0-250-109.us-west-2.compute.internal uid/16380fbc-f9d7-4f62-86f2-258ad4402448 container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 06 14:16:15.880 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-123542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-b3c1e99ee573760d7324a763e1b578b3 expected: registry.ci.openshift.org/ocp/4.12-2024-05-06-123542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-03-193550@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]
May 06 14:16:15.880 - 359s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-123542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-b3c1e99ee573760d7324a763e1b578b3 expected: registry.ci.openshift.org/ocp/4.12-2024-05-06-123542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-03-193550@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]

... 1 lines not shown

May 06 20:36:16.792 E ns/openshift-multus pod/multus-additional-cni-plugins-vkm6b node/ip-10-0-170-95.us-west-2.compute.internal uid/ddfdfd14-39e7-44aa-93d8-a32be2d85363 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 06 20:36:37.701 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-f4d65436176fb560e55450fc98a5d780 expected: registry.ci.openshift.org/ocp/4.12-2024-05-06-183542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-06-123542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]
May 06 20:36:37.701 - 304s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-f4d65436176fb560e55450fc98a5d780 expected: registry.ci.openshift.org/ocp/4.12-2024-05-06-183542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-06-123542@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]

... 1 lines not shown

May 06 12:54:33.790 E ns/openshift-dns pod/dns-default-g4zm5 node/ip-10-0-177-137.ec2.internal uid/b4fe883c-5060-49d8-aaa4-448ad9f6ac48 container/kube-rbac-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 06 12:54:34.207 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-c483b5d79df2b13992d1456eb349fbd8 expected: registry.ci.openshift.org/ocp/4.12-2024-05-03-193550@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-03-102448@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]
May 06 12:54:34.207 - 338s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-c483b5d79df2b13992d1456eb349fbd8 expected: registry.ci.openshift.org/ocp/4.12-2024-05-03-193550@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e got: registry.ci.openshift.org/ocp/4.12-2024-05-03-102448@sha256:5d6510261ba9a484c8797299e10a4b31be6ccdc400b238aebae2702aec52bc3e, retrying]

... 1 lines not shown