May 02 01:25:46.117 E ns/openshift-multus pod/multus-admission-controller-7bbd89c4c8-2gqg5 node/ip-10-0-134-118.us-east-2.compute.internal uid/580a14af-68cc-4bc5-b0fa-190034ad00c8 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 02 01:28:50.999 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-48dec65b4a5e37021d63125daeb0664e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-839d3f9b6f2aab727c4d0f8bd79069aa, retrying]
May 02 01:28:50.999 - 161s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-48dec65b4a5e37021d63125daeb0664e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-839d3f9b6f2aab727c4d0f8bd79069aa, retrying]

... 1 lines not shown

May 01 22:18:10.161 E ns/e2e-k8s-sig-apps-job-upgrade-9917 pod/foo-tqzfh node/ip-10-0-253-228.us-east-2.compute.internal uid/bf6dde40-c1aa-43d5-881d-efd369c652c7 container/c reason/ContainerExit code/137 cause/Error
May 01 22:18:45.512 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6d1c4eae4b05813e3680cc18057dc6dd expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a7dcb40a8689800c5ca06dbccd4bb45e, retrying]
May 01 22:18:45.512 - 644s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6d1c4eae4b05813e3680cc18057dc6dd expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a7dcb40a8689800c5ca06dbccd4bb45e, retrying]

... 1 lines not shown

May 01 19:16:48.836 E ns/openshift-multus pod/multus-admission-controller-7bbd89c4c8-s7dzh node/ip-10-0-164-180.us-west-2.compute.internal uid/25d569f4-075b-490c-9275-0e66afaf0371 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 01 19:19:57.782 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-948352f381485180a5863bd9c1e5341c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ce17e0dc6c29ee55f760c2d2a2819baa, retrying]
May 01 19:19:57.782 - 152s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-948352f381485180a5863bd9c1e5341c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ce17e0dc6c29ee55f760c2d2a2819baa, retrying]

... 1 lines not shown

May 01 05:13:05.650 E ns/e2e-k8s-sig-apps-job-upgrade-653 pod/foo-fkrpq node/ip-10-0-216-15.us-west-2.compute.internal uid/a226bf26-3056-447f-8275-a1a33623201d container/c reason/ContainerExit code/137 cause/Error
May 01 05:14:47.117 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-031456: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bce09a51481f8b3b01fd52fbecb834b6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ae410536447b75962dd5eb97f5752ce8, retrying]
May 01 05:14:47.117 - 548s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-031456: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bce09a51481f8b3b01fd52fbecb834b6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ae410536447b75962dd5eb97f5752ce8, retrying]

... 1 lines not shown

Apr 30 22:45:06.258 E ns/e2e-k8s-sig-apps-job-upgrade-8707 pod/foo-jwwlv node/ip-10-0-144-208.us-west-1.compute.internal uid/34d5fbf2-1206-4596-839c-c122e0a739c3 container/c reason/ContainerExit code/137 cause/Error
Apr 30 22:45:34.769 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-11f47f7005ed1dc91ef2cd83ab03a5e6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-03660e08d98e10c46178da35a12cf503, retrying]
Apr 30 22:45:34.769 - 560s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-11f47f7005ed1dc91ef2cd83ab03a5e6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-03660e08d98e10c46178da35a12cf503, retrying]

... 1 lines not shown

Apr 30 19:50:58.842 E ns/openshift-multus pod/multus-admission-controller-97c556c7d-km84p node/ip-10-0-197-112.us-east-2.compute.internal uid/152bc653-5396-4e2c-986d-973bfe636e7d container/multus-admission-controller reason/ContainerExit code/137 cause/Error
Apr 30 19:54:23.820 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3fd5b9721a97742fd2c1c4c24547044b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2f6e47806671fbf34d732615e6a4d81f, retrying]
Apr 30 19:54:23.820 - 129s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3fd5b9721a97742fd2c1c4c24547044b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2f6e47806671fbf34d732615e6a4d81f, retrying]

... 1 lines not shown

Apr 30 16:32:23.257 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-158-155.us-west-2.compute.internal uid/b25eacdf-380c-4f86-9389-1f28fe62299d container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-04-30T15:57:34.584825154Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-04-30T15:57:34.584890666Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-04-30T15:57:34.585049409Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-04-30T15:57:38.656683481Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-04-30T15:57:38.656774913Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-04-30T15:57:51.728505104Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-04-30T16:00:30.440821209Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-04-30T16:04:05.057915352Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles
Apr 30 16:32:34.557 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-906c65af4fcd65eb8afea7ea0a95d1a2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-13347407b7aed8be476d73fb555a4c62, retrying]
Apr 30 16:32:34.557 - 563s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-906c65af4fcd65eb8afea7ea0a95d1a2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-13347407b7aed8be476d73fb555a4c62, retrying]

... 1 lines not shown

Apr 30 13:18:43.614 E ns/e2e-k8s-sig-apps-job-upgrade-1319 pod/foo-m7slh node/ip-10-0-249-156.us-west-2.compute.internal uid/d33055f9-5b52-4183-a770-f94964384afb container/c reason/ContainerExit code/137 cause/Error
Apr 30 13:19:35.586 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6803c8d64cd4843e6ecfbf2a08eda3b6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-efe5f469cca0b6b580a87990e96a25f2, retrying]
Apr 30 13:19:35.586 - 579s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6803c8d64cd4843e6ecfbf2a08eda3b6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-efe5f469cca0b6b580a87990e96a25f2, retrying]

... 1 lines not shown

May 01 23:36:37.564 E ns/e2e-k8s-sig-apps-job-upgrade-1258 pod/foo-f7lc5 node/ip-10-0-165-77.ec2.internal uid/d22f5664-dee1-42d8-bac7-55c0fc778e5a container/c reason/ContainerExit code/137 cause/Error
May 01 23:38:09.201 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.11.59: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9ffcde83b49141e2f528b7d70a22281d expected 15d0b0288a4330b90ac89f14c781dfa7349af52c has 59a8d308bfa4cd89abdec9d345e396fe9dafd7b0: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-bcd67ef94e427fbd003a6d57d70a77fe, retrying]
May 01 23:38:09.201 - 473s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.11.59: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9ffcde83b49141e2f528b7d70a22281d expected 15d0b0288a4330b90ac89f14c781dfa7349af52c has 59a8d308bfa4cd89abdec9d345e396fe9dafd7b0: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-bcd67ef94e427fbd003a6d57d70a77fe, retrying]

... 1 lines not shown

May 01 22:50:19.298 E ns/e2e-k8s-sig-apps-job-upgrade-53 pod/foo-l9zqm node/ip-10-0-251-105.us-west-2.compute.internal uid/3826081a-2c59-41de-acb0-4ddcc6f96b05 container/c reason/ContainerExit code/137 cause/Error
May 01 22:52:49.142 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-01-172252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5c749eb7d4c86e75489a6512bbb69c84 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b93342f5d2e7b81fa8afe734ace0985f, retrying]
May 01 22:52:49.142 - 538s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-01-172252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5c749eb7d4c86e75489a6512bbb69c84 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b93342f5d2e7b81fa8afe734ace0985f, retrying]

... 1 lines not shown

May 01 19:18:46.011 - 2s    E clusteroperator/storage condition/Degraded status/True reason/AWSEBSCSIDriverOperatorDegraded: Get "https://172.30.0.1:443/apis/operator.openshift.io/v1/clustercsidrivers/ebs.csi.aws.com": context canceled
May 01 19:20:29.889 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-01-172252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ddb280555edff937732b40c4d5bdfa23 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f9c3d6d1ccff7bb0a5797db062899fa2, retrying]
May 01 19:20:29.889 - 479s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-01-172252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ddb280555edff937732b40c4d5bdfa23 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f9c3d6d1ccff7bb0a5797db062899fa2, retrying]

... 1 lines not shown

Apr 30 14:23:44.288 - 102ms E clusteroperator/storage condition/Degraded status/True reason/AWSEBSCSIDriverOperatorDeploymentDegraded: Get "https://172.30.0.1:443/apis/apps/v1/namespaces/openshift-cluster-csi-drivers/deployments/aws-ebs-csi-driver-operator": context canceled
Apr 30 14:24:33.746 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-04-30-123839: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a7e871294d671a8b86990ffaf295bc19 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-53ab1dc94f4ddd9fbb1a8a32f2a75dbd, retrying]
Apr 30 14:24:33.746 - 148s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-04-30-123839: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a7e871294d671a8b86990ffaf295bc19 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-53ab1dc94f4ddd9fbb1a8a32f2a75dbd, retrying]

... 1 lines not shown

Apr 30 11:22:26.877 E ns/e2e-k8s-sig-apps-job-upgrade-2035 pod/foo-k4n86 node/ip-10-0-177-245.us-west-2.compute.internal uid/ffd2b128-c043-42c1-9a98-0829430a7f4b container/c reason/ContainerExit code/137 cause/Error
Apr 30 11:22:37.313 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-04-30-093127: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-825c244648734a3546de126c5aa31fb8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1735d7ec3773b3b0a51a059c3491be1c, retrying]
Apr 30 11:22:37.313 - 131s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-04-30-093127: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-825c244648734a3546de126c5aa31fb8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1735d7ec3773b3b0a51a059c3491be1c, retrying]

... 1 lines not shown

Apr 30 07:29:00.205 - 1s    E node/ip-10-0-204-65.ec2.internal reason/FailedToDeleteCGroupsPath Apr 30 07:29:00.205186 ip-10-0-204-65 hyperkube[1382]: I0430 07:29:00.205156    1382 pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods burstable podb3c89e24-3c1a-430f-9796-188e74fa0325] err="unable to destroy cgroup paths for cgroup [kubepods burstable podb3c89e24-3c1a-430f-9796-188e74fa0325] : Timed out while waiting for systemd to remove kubepods-burstable-podb3c89e24_3c1a_430f_9796_188e74fa0325.slice"
Apr 30 07:30:17.497 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-04-30-055344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-66d23653f1050b00da367505a34ae42f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7a5099a75d36b6088d60874962a61139, retrying]
Apr 30 07:30:17.497 - 492s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-04-30-055344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-66d23653f1050b00da367505a34ae42f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7a5099a75d36b6088d60874962a61139, retrying]

... 1 lines not shown

May 01 21:31:39.036 E ns/openshift-monitoring pod/alertmanager-main-0 node/libvirt-ppc64le-1-2-e-5xrhl-worker-0-hq47m uid/bc9b3c24-67cb-4008-95a5-80b3410f3c6f container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-01T21:31:37.100Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=c5f095c646ea4c1f9524833e8e3579b827a501aa)"\nts=2024-05-01T21:31:37.100Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@0a12bfb51770, date=20240425-19:11:18)"\nts=2024-05-01T21:31:37.136Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-01T21:31:37.180Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-01T21:31:37.180Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-01T21:31:37.180Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=43.941341ms\n
May 01 21:33:04.999 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-ppc64le-2024-05-01-030927: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-244be66eac9a4134ae158734187b8ea8 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b9d6fd047b6aa8c8b88de0c16c36962a, retrying]
May 01 21:33:04.999 - 541s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-ppc64le-2024-05-01-030927: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-244be66eac9a4134ae158734187b8ea8 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b9d6fd047b6aa8c8b88de0c16c36962a, retrying]

... 1 lines not shown

May 01 09:43:14.460 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-pd8hh node/libvirt-ppc64le-2-1-e-vwwnl-worker-0-tksbc uid/b7755876-5060-45c2-8297-c22f5e4e277c container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 01 09:46:38.641 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-ppc64le-2024-05-01-030927: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c15dc3f0ebca4e4cba89af3a05db4994 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8a73dd8595ae2e661b1f4361d5c7fabd, retrying]
May 01 09:46:38.641 - 97s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-ppc64le-2024-05-01-030927: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c15dc3f0ebca4e4cba89af3a05db4994 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8a73dd8595ae2e661b1f4361d5c7fabd, retrying]

... 1 lines not shown

Apr 30 21:52:00.278 E ns/e2e-check-for-dns-availability-5180 pod/dns-test-c3eb0011-95e3-4d80-8713-9a797a72010b-fw9cm node/libvirt-ppc64le-0-1-e-4tng9-worker-0-xrrr2 uid/fa7df38b-8c25-4180-9bce-cb32d099dd63 container/querier reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 21:53:41.729 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-ppc64le-2024-04-30-192537: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-50a09e973e202670a80316216ffee1d8 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d26d5d84c05cc364af483d1df6b510e9, retrying]
Apr 30 21:53:41.729 - 475s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-ppc64le-2024-04-30-192537: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-50a09e973e202670a80316216ffee1d8 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d26d5d84c05cc364af483d1df6b510e9, retrying]

... 1 lines not shown

Apr 30 09:40:20.596 E ns/openshift-dns pod/dns-default-k9w4f node/libvirt-ppc64le-1-1-e-xhfqr-worker-0-f6xt4 uid/ff889678-04ef-4a43-807e-9d806f0d0696 container/kube-rbac-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 09:42:13.726 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-ppc64le-2024-04-29-100621: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-90fb161718333ef7e5d84714de96078f expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-938847b74a21213cd9b189cc930cfe18, retrying]
Apr 30 09:42:13.726 - 556s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-ppc64le-2024-04-29-100621: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-90fb161718333ef7e5d84714de96078f expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-938847b74a21213cd9b189cc930cfe18, retrying]

... 1 lines not shown

May 01 18:40:47.862 - 121s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.56: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6cdebf9f425732efbdea341f5c5d392f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1083fb3b5bde780780202c4648fed740, retrying]
May 01 19:45:55.185 - 121s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1083fb3b5bde780780202c4648fed740 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5a91bcc0523479524e362b102c714757, retrying]

... 1 lines not shown

Apr 30 18:39:49.105 - 504s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.56: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-777e076f7b616fc172f56bafd57331a8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-40afa63985e2fb3c55a449f23f6724e2, retrying]
1 tests failed during this blip (2024-04-30 18:39:49.105654225 +0000 UTC to 2024-04-30 18:39:49.105654225 +0000 UTC): [sig-arch][Feature:ClusterUpgrade] Cluster should remain functional during upgrade [Disruptive] [Serial]

... 2 lines not shown

May 01 19:11:01.291 E ns/openshift-e2e-loki pod/loki-promtail-zncqv node/ci-op-y4hqhkmm-7a89c-dbdns-worker-c-t8kp9 uid/775edbc5-80f7-4428-81ba-a5e174372a74 container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 19:12:15.329 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-01-172206: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4909bc464276bc7fb93723eaa7aab900 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-354604cdb38b4d561b7fc02b87065ac3, retrying]
May 01 19:12:15.329 - 114s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-01-172206: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4909bc464276bc7fb93723eaa7aab900 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-354604cdb38b4d561b7fc02b87065ac3, retrying]

... 1 lines not shown

May 01 07:53:56.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://34.173.31.154:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 01 07:54:02.917 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-01-055836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-648d6a3613c38025920b99e08df59878 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f13fca461ecade6fee6c7bc9ed48681c, retrying]
May 01 07:54:02.917 - 95s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-01-055836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-648d6a3613c38025920b99e08df59878 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f13fca461ecade6fee6c7bc9ed48681c, retrying]

... 1 lines not shown

May 01 01:52:06.674 E ns/openshift-e2e-loki pod/loki-promtail-l6gp4 node/ci-op-g8rng2bm-7a89c-xz65v-worker-c-rghd6 uid/22ac66d7-6b6c-45c0-a371-5efec8e5b69b container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 01:53:25.762 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-235836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2ac2d19077f614ed25403d94afca65db expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2a476ef2b2e2503b90e3abd0f4886fe9, retrying]
May 01 01:53:25.762 - 90s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-235836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2ac2d19077f614ed25403d94afca65db expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2a476ef2b2e2503b90e3abd0f4886fe9, retrying]

... 1 lines not shown

Apr 30 17:46:33.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://35.225.132.46:80/echo?msg=Hello": net/http: timeout awaiting response headers
Apr 30 17:46:44.135 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-155425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-748d2456cd4ec0b89863b0bbc8c6611b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f0c0897c36c2fdebba3f7b40a7e5ecbc, retrying]
Apr 30 17:46:44.135 - 85s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-155425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-748d2456cd4ec0b89863b0bbc8c6611b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f0c0897c36c2fdebba3f7b40a7e5ecbc, retrying]

... 1 lines not shown

Apr 30 11:14:55.559 E ns/openshift-e2e-loki pod/loki-promtail-68vzn node/ci-op-0x03f6bh-7a89c-tpzsc-worker-c-hcj2c uid/9dc3f3b9-86d8-4fc6-8b6a-15435b48530b container/prod-bearer-token reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 11:16:04.060 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-093053: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-428ea32c56662d6c02f938ce87d010f8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0893ccee8b0446752ca8a996f00cd5d3, retrying]
Apr 30 11:16:04.060 - 140s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-093053: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-428ea32c56662d6c02f938ce87d010f8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0893ccee8b0446752ca8a996f00cd5d3, retrying]

... 1 lines not shown

May 01 19:00:15.146 E ns/e2e-k8s-sig-apps-job-upgrade-8392 pod/foo-r6f64 node/ip-10-0-232-228.us-east-2.compute.internal uid/c079c83f-46ec-4bd8-9bc2-a1c1e9c65f38 container/c reason/ContainerExit code/137 cause/Error
May 01 19:02:41.408 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-689441cc3c7342639508bfd5b582f4de expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-aaefd6e128907a819446253dab45ae47, retrying]
May 01 19:02:41.408 - 463s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-172823: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-689441cc3c7342639508bfd5b582f4de expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-aaefd6e128907a819446253dab45ae47, retrying]

... 1 lines not shown

May 01 13:49:07.000 - 1s    E disruption/service-load-balancer-with-pdb connection/new reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/new stopped responding to GET requests over new connections: Get "http://aced128a088f74a2db71c7bd51b9f32e-1936139667.us-west-1.elb.amazonaws.com:80/echo?msg=Hello": read tcp 10.131.73.149:41386->52.52.203.48:80: read: connection reset by peer
May 01 13:49:41.675 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-031456: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2c8303aed6912d6a8df955bfddab797 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c4f50a4790e8f8485f997b9cf3ec9ecd, retrying]
May 01 13:49:41.675 - 486s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-031456: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2c8303aed6912d6a8df955bfddab797 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c4f50a4790e8f8485f997b9cf3ec9ecd, retrying]

... 1 lines not shown

May 01 05:03:07.569 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-tlvdp node/ip-10-0-145-203.us-west-2.compute.internal uid/c2897a34-e0ac-4844-a277-823716088a7a container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 01 05:06:07.429 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-031456: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d85a39c4b4d8fc622b1ef7e74e878299 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-85e346b5b275400f6971bfcbe63c9c40, retrying]
May 01 05:06:07.429 - 128s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-031456: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d85a39c4b4d8fc622b1ef7e74e878299 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-85e346b5b275400f6971bfcbe63c9c40, retrying]

... 1 lines not shown

Apr 30 22:39:40.368 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-185-182.us-west-2.compute.internal uid/35b3a251-0f25-4d58-b2c0-dc8512f0b1f2 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-04-30T22:39:36.977Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=release-4.12, revision=914cad827e9a177b29b23e02eb48b4065da8dca2)"\nts=2024-04-30T22:39:36.977Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@06d38c1ed95c, date=20231101-03:59:15)"\nts=2024-04-30T22:39:37.007Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-04-30T22:39:37.036Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-04-30T22:39:37.036Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-04-30T22:39:37.036Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=28.970009ms\n
Apr 30 22:41:25.761 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-098788c4b20ed820e04ad3a4237bb270 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-9130540025fc7e553502edf526cfa2a6, retrying]
Apr 30 22:41:25.761 - 503s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-098788c4b20ed820e04ad3a4237bb270 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-9130540025fc7e553502edf526cfa2a6, retrying]

... 1 lines not shown

Apr 30 13:14:43.092 E ns/openshift-multus pod/multus-admission-controller-6b9477f8f7-dlfvm node/ip-10-0-142-76.us-west-1.compute.internal uid/cafb5f37-3308-4f0a-b0d5-c7366e620055 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
Apr 30 13:17:46.984 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1411a1579ad1aa6b55194dbbf0844aad expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-222310178951f0a5de1c1f278913812f, retrying]
Apr 30 13:17:46.984 - 135s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1411a1579ad1aa6b55194dbbf0844aad expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-222310178951f0a5de1c1f278913812f, retrying]

... 1 lines not shown

May 01 20:08:33.632 E ns/e2e-check-for-dns-availability-7944 pod/dns-test-2f84e7ec-8fe4-469b-9721-52897eadcfe0-2526q node/worker-1.ostest.test.metalkube.org uid/d18e886f-ffc2-4f04-a99f-cc20bc00764f container/querier reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 20:08:40.107 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-01-172206: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bfd905c52fe2c2cf6c8ea7bf1325d0a1 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-898de905bac23a1f981ee4f1cfed3611, retrying]
May 01 20:08:40.107 - 203s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-01-172206: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bfd905c52fe2c2cf6c8ea7bf1325d0a1 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-898de905bac23a1f981ee4f1cfed3611, retrying]

... 1 lines not shown

May 01 03:01:05.845 E ns/openshift-multus pod/network-metrics-daemon-bnsxh node/worker-1.ostest.test.metalkube.org uid/7fec440f-9255-4c60-bb7a-9c6027bff259 container/kube-rbac-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 03:01:17.648 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-235836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c7a8d9bfc75518987415de69a6e14db8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-185dd1558f120a7bdd5b170893536405, retrying]
May 01 03:01:17.648 - 172s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-235836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c7a8d9bfc75518987415de69a6e14db8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-185dd1558f120a7bdd5b170893536405, retrying]

... 1 lines not shown

Apr 30 18:37:55.412 - 999ms E disruption/cache-oauth-api connection/reused reason/DisruptionBegan disruption/cache-oauth-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ostest.test.metalkube.org:6443/apis/oauth.openshift.io/v1/oauthclients?resourceVersion=0": net/http: timeout awaiting response headers
Apr 30 18:39:51.987 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-155425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3e6787e417597c23c8c2c326b75d2924 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9f24017e3fa1642842755682acb75a20, retrying]
Apr 30 18:39:51.987 - 190s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-155425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3e6787e417597c23c8c2c326b75d2924 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9f24017e3fa1642842755682acb75a20, retrying]

... 1 lines not shown

Apr 30 12:18:17.640 E ns/openshift-multus pod/multus-additional-cni-plugins-dd6nz node/worker-1.ostest.test.metalkube.org uid/0af62cd1-f16a-437f-97fd-b2e4a15e106b container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 12:18:24.391 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-093053: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-988116236258dc80680cdf520a5bc457 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-76b5ec1c82a3c2b4f050f1fa18520965, retrying]
Apr 30 12:18:24.391 - 236s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-093053: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-988116236258dc80680cdf520a5bc457 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-76b5ec1c82a3c2b4f050f1fa18520965, retrying]

... 1 lines not shown

May 01 18:55:23.315 E ns/e2e-k8s-sig-apps-job-upgrade-9442 pod/foo-cdd5h node/ip-10-0-171-69.us-west-2.compute.internal uid/d0084723-2b08-40a0-9942-b02188aed7f4 container/c reason/ContainerExit code/137 cause/Error
May 01 18:56:23.216 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-01-172206: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-12100be0b41240c60cd049c643a06aea expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3911e4d88716f363efece5f9f72774c1, retrying]
May 01 18:56:23.216 - 519s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-01-172206: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-12100be0b41240c60cd049c643a06aea expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3911e4d88716f363efece5f9f72774c1, retrying]

... 1 lines not shown

May 01 07:36:57.696 - 35s   E clusteroperator/network condition/Degraded status/True reason/Error while updating operator configuration: could not apply (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus-admission-controller-webhook: failed to apply / update (rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding) /multus-admission-controller-webhook: Patch "https://api-int.ci-op-l7v5c6lc-857a8.aws-2.ci.openshift.org:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings/multus-admission-controller-webhook?fieldManager=cluster-network-operator%2Foperconfig&force=true": read tcp 10.0.230.161:45700->10.0.131.34:6443: read: connection reset by peer
May 01 07:37:28.000 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-01-055836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e1f9f8e93422e25d3fc96b7896feffe4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fd695b3198bca272679239f77c2a1af8, retrying]
May 01 07:37:28.000 - 522s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-01-055836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e1f9f8e93422e25d3fc96b7896feffe4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fd695b3198bca272679239f77c2a1af8, retrying]

... 1 lines not shown

May 01 01:30:46.025 E ns/e2e-k8s-sig-apps-job-upgrade-3111 pod/foo-fpjhx node/ip-10-0-199-207.ec2.internal uid/d943b0b9-6af2-4994-a306-87cce9f61484 container/c reason/ContainerExit code/137 cause/Error
May 01 01:32:27.364 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-235836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4d39a64e94fb42056e1ae59e2e35849e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-530f784bd286c7bf808770e153f3f642, retrying]
May 01 01:32:27.364 - 511s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-235836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4d39a64e94fb42056e1ae59e2e35849e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-530f784bd286c7bf808770e153f3f642, retrying]

... 1 lines not shown

Apr 30 17:30:13.578 E ns/openshift-multus pod/multus-admission-controller-84cc4444c4-ssd4s node/ip-10-0-239-69.us-west-2.compute.internal uid/a3c4b10d-8add-497f-b991-87e84814d18d container/multus-admission-controller reason/ContainerExit code/137 cause/Error
Apr 30 17:33:13.782 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-155425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-00e06f6c8e5bfce7a666b57fb0a3f651 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6c7cc3dd4d3486b8aeeadb271d994751, retrying]
Apr 30 17:33:13.782 - 145s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-155425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-00e06f6c8e5bfce7a666b57fb0a3f651 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6c7cc3dd4d3486b8aeeadb271d994751, retrying]

... 1 lines not shown

Apr 30 11:06:06.004 E ns/openshift-multus pod/multus-admission-controller-84cc4444c4-bh9lh node/ip-10-0-240-143.ec2.internal uid/298aa52f-9830-4a0c-be54-1e6d3cbf32be container/multus-admission-controller reason/ContainerExit code/137 cause/Error
Apr 30 11:09:44.492 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-093053: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a0fccf68abe7fc5857c0d71f6be74ec5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-538d6352a80a499858ae9f6822e0b104, retrying]
Apr 30 11:09:44.492 - 86s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-093053: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a0fccf68abe7fc5857c0d71f6be74ec5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-538d6352a80a499858ae9f6822e0b104, retrying]

... 1 lines not shown

May 01 19:03:14.080 - 999ms E ns/openshift-authentication route/oauth-openshift disruption/ingress-to-oauth-server connection/new reason/DisruptionBegan ns/openshift-authentication route/oauth-openshift disruption/ingress-to-oauth-server connection/new stopped responding to GET requests over new connections: Get "https://oauth-openshift.apps.ci-op-74fgl0ht-20826.aws-2.ci.openshift.org/healthz": read tcp 10.128.154.3:59760->52.8.211.218:443: read: connection reset by peer
May 01 19:03:28.289 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-01-172210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bbff268832f1d0f2d3ab962035adc73b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-aa08ca6de78c5841fd926d4dfa2870c6, retrying]
May 01 19:03:28.289 - 112s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-01-172210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bbff268832f1d0f2d3ab962035adc73b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-aa08ca6de78c5841fd926d4dfa2870c6, retrying]

... 1 lines not shown

May 01 04:16:06.000 - 1s    E ns/openshift-image-registry route/test-disruption-new disruption/image-registry connection/new reason/DisruptionBegan ns/openshift-image-registry route/test-disruption-new disruption/image-registry connection/new stopped responding to GET requests over new connections: Get "https://test-disruption-new-openshift-image-registry.apps.ci-op-03b56p8h-20826.aws-2.ci.openshift.org/healthz": read tcp 10.131.76.97:52984->44.227.50.14:443: read: connection reset by peer
May 01 04:16:22.639 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-01-024018: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1981a0504f413bf01b3a6bd02c92afc2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-8a2a390193a29cd9e2129fbf80680ef4, retrying]
May 01 04:16:22.639 - 503s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-01-024018: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1981a0504f413bf01b3a6bd02c92afc2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-8a2a390193a29cd9e2129fbf80680ef4, retrying]

... 1 lines not shown

Apr 30 11:01:02.223 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-xpjhc node/ip-10-0-197-111.us-east-2.compute.internal uid/daf0dd48-8714-4aa4-bb92-3e070316ba4e container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
Apr 30 11:03:57.693 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-04-30-093054: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14681c5e6da62e2e3bf8e78d22534172 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1b72f1f245775a9bf711d53dc5c6af43, retrying]
Apr 30 11:03:57.693 - 72s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-04-30-093054: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14681c5e6da62e2e3bf8e78d22534172 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1b72f1f245775a9bf711d53dc5c6af43, retrying]

... 1 lines not shown

May 01 19:49:46.711 E ns/openshift-kni-infra pod/haproxy-host2.cluster16.ocpci.eng.rdu2.redhat.com node/host2.cluster16.ocpci.eng.rdu2.redhat.com uid/f69662e2-acd2-44b9-9743-55235e39be29 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 19:51:19.613 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-01-172206: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4a69ebfd13d21667d7de6a1887dd6eae expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-224f8de321e98c7aac0b464cccf56f3a, retrying]
May 01 19:51:19.613 - 123s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-01-172206: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4a69ebfd13d21667d7de6a1887dd6eae expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-224f8de321e98c7aac0b464cccf56f3a, retrying]

... 1 lines not shown

May 01 08:22:11.511 E ns/openshift-dns pod/dns-default-8tdwd node/host5.cluster6.ocpci.eng.rdu2.redhat.com uid/9fe5a669-2768-4f8c-ba56-c07272bf4263 container/dns reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 08:22:11.609 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-01-055836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-03dd4e666de4b90d0192d770a1b2bea2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e25981ef6f9b0d1911de399cdc30fee9, retrying]
May 01 08:22:11.609 - 501s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-01-055836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-03dd4e666de4b90d0192d770a1b2bea2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e25981ef6f9b0d1911de399cdc30fee9, retrying]

... 1 lines not shown

Apr 30 18:33:12.518 E ns/openshift-kni-infra pod/haproxy-host3.cluster15.ocpci.eng.rdu2.redhat.com node/host3.cluster15.ocpci.eng.rdu2.redhat.com uid/bf60bd91-15e8-4928-b8bd-c82735a17896 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 18:34:52.680 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-155425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ea13738743bea87947f2e9ed08b8c0d5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c63cf676edff430a2326a554ba44d634, retrying]
Apr 30 18:34:52.680 - 491s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-155425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ea13738743bea87947f2e9ed08b8c0d5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c63cf676edff430a2326a554ba44d634, retrying]

... 1 lines not shown

Apr 30 11:55:40.182 E ns/openshift-monitoring pod/node-exporter-wrpcm node/host5.cluster11.ocpci.eng.rdu2.redhat.com uid/75a37839-d12d-4a44-a7b2-81447e841be6 container/node-exporter reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 11:55:40.774 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-093053: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4e3b996e461675bc0fa9259e112c299b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-aa12a91ff94e07d5e3113619c4bc4f32, retrying]
Apr 30 11:55:40.774 - 546s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-093053: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4e3b996e461675bc0fa9259e112c299b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-aa12a91ff94e07d5e3113619c4bc4f32, retrying]

... 1 lines not shown

May 01 13:58:29.770 - 209ms E clusteroperator/storage condition/Degraded status/True reason/AWSEBSCSIDriverOperatorDeploymentDegraded: Get "https://172.30.0.1:443/apis/apps/v1/namespaces/openshift-cluster-csi-drivers/deployments/aws-ebs-csi-driver-operator": context canceled
May 01 13:59:08.968 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-multi-2024-05-01-024052: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f978208e491481574c1f2b116ff97b25 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-98482ef088453ceba180fc3898fc9e4b, retrying]
May 01 13:59:08.968 - 540s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-multi-2024-05-01-024052: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f978208e491481574c1f2b116ff97b25 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-98482ef088453ceba180fc3898fc9e4b, retrying]

... 1 lines not shown

May 01 13:25:13.774 E ns/openshift-multus pod/multus-admission-controller-84cc4444c4-qb9lj node/ip-10-0-177-70.us-west-2.compute.internal uid/2b6add73-801c-4e4e-a9c2-9779d90774fe container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 01 13:28:54.306 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-04-30-235836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1f269bb29aac924c58c9f1b3e239a02e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9e2d5d8901e5a00cdfb3058dcca0aced, retrying]
May 01 13:28:54.306 - 117s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-04-30-235836: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1f269bb29aac924c58c9f1b3e239a02e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9e2d5d8901e5a00cdfb3058dcca0aced, retrying]

... 1 lines not shown

May 01 12:00:16.747 - 1s    E disruption/openshift-api connection/reused reason/DisruptionBegan disruption/openshift-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ci-op-hd91x4t5-7cadb.ci2.azure.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/default/imagestreams": net/http: timeout awaiting response headers
May 01 12:02:34.413 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-01-064657: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c7a02b7dec5130e55c31f05dfd30d047 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5b89d85b75ae98c37c4d236328a1300d, retrying]
May 01 12:02:34.413 - 194s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-01-064657: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c7a02b7dec5130e55c31f05dfd30d047 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5b89d85b75ae98c37c4d236328a1300d, retrying]

... 1 lines not shown

May 01 05:19:48.322 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-k4zt4 node/ci-op-3sq9ywpz-7cadb-htj5c-worker-westus-xc8xb uid/35d6cf4f-3e26-49d7-9e21-1da363cc8cc8 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 01 05:22:07.768 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-220027: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9ce5978551a35ee555b2ae2feba1ee54 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4d952ba37c011130ea648705c6b4ebd3, retrying]
May 01 05:22:07.768 - 194s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-220027: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9ce5978551a35ee555b2ae2feba1ee54 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4d952ba37c011130ea648705c6b4ebd3, retrying]

... 1 lines not shown

Apr 30 23:57:08.157 E ns/openshift-multus pod/multus-additional-cni-plugins-vjfjp node/ci-op-1qblmn2y-7cadb-mkqcx-worker-westus-wzw6r uid/7bcd1016-6a46-4d35-9535-c24710bc9dbd container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 23:57:24.380 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-220027: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eecb607a40c098b5989f55d8a6c9ffb3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a3ca525b916ff370daef6f7f640ab181, retrying]
Apr 30 23:57:24.380 - 707s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-220027: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eecb607a40c098b5989f55d8a6c9ffb3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a3ca525b916ff370daef6f7f640ab181, retrying]

... 1 lines not shown

Apr 30 20:43:07.049 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-yv8r5l9f-7cadb-2pxtc-worker-centralus3-m2lx6 uid/f5ef6010-0860-4ae6-a8dc-9a9b1b91db97 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-04-30T20:16:52.650897456Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.63.0, branch=release-4.13, revision=30fdccd13)"\nlevel=info ts=2024-04-30T20:16:52.651003056Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/amd64, user=root, date=20231108-08:14:29)"\nlevel=info ts=2024-04-30T20:16:52.651408858Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-04-30T20:17:02.517680073Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-04-30T20:17:02.517847373Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-04-30T20:17:07.509289475Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-04-30T20:18:08.889237659Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-04-30T20:19:39.424231336Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/pr
Apr 30 20:44:15.193 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a0975d6d2bdebd7f6c86fd3427b141c7 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d282b4bceefc9410dac688e91d53cbed, retrying]
Apr 30 20:44:15.193 - 482s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a0975d6d2bdebd7f6c86fd3427b141c7 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d282b4bceefc9410dac688e91d53cbed, retrying]

... 1 lines not shown

Apr 30 17:42:05.037 E ns/e2e-k8s-sig-apps-job-upgrade-7891 pod/foo-hnstn node/ci-op-wmv502qi-7cadb-9jmz8-worker-centralus3-d9fpv uid/2dd2d5f2-2388-4a3f-b11d-9126277345ca container/c reason/ContainerExit code/137 cause/Error
Apr 30 17:42:18.838 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-81cd4fc861f92077706f53ccf9688627 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ebc86c98beb8ca1032e7c244dd81cf3a, retrying]
Apr 30 17:42:18.838 - 565s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-81cd4fc861f92077706f53ccf9688627 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ebc86c98beb8ca1032e7c244dd81cf3a, retrying]

... 1 lines not shown

May 01 10:41:17.387 E ns/e2e-k8s-sig-apps-job-upgrade-5250 pod/foo-zbx5f node/ci-op-frn9brjv-3a480-rj4pw-worker-b-h4zlq uid/f2dba975-3c45-4169-bc61-e04eed529aa2 container/c reason/ContainerExit code/137 cause/Error
May 01 10:41:53.856 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-98eee9b0caa144be149b6947af7b2e0a expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-67a3b7eab1dc5e34874054435f98ba87, retrying]
May 01 10:41:53.856 - 613s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-98eee9b0caa144be149b6947af7b2e0a expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-67a3b7eab1dc5e34874054435f98ba87, retrying]

... 1 lines not shown

May 01 07:13:47.955 E ns/openshift-monitoring pod/openshift-state-metrics-69b8c7f498-wb2dx node/ci-op-qbglc79x-3a480-75wrs-worker-b-w72l4 uid/1044a300-1ccd-46c4-bb1e-1c528b7defee container/openshift-state-metrics reason/ContainerExit code/2 cause/Error
May 01 07:14:11.745 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-41caed0aca6fa98ba4003197e06da9f0 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-46a33314598cab849ce81d9a31273895, retrying]
May 01 07:14:11.745 - 623s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-41caed0aca6fa98ba4003197e06da9f0 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-46a33314598cab849ce81d9a31273895, retrying]

... 1 lines not shown

May 01 01:25:05.072 - 1s    E node/ci-op-z22fxxnx-3a480-tz6rx-master-2 reason/FailedToDeleteCGroupsPath May 01 01:25:05.072132 ci-op-z22fxxnx-3a480-tz6rx-master-2 kubenswrapper[2211]: I0501 01:25:05.070881    2211 pod_container_manager_linux.go:191] "Failed to delete cgroup paths" cgroupName=[kubepods burstable poda3d6837195b14fb3d98dd6983a993c29] err="unable to destroy cgroup paths for cgroup [kubepods burstable poda3d6837195b14fb3d98dd6983a993c29] : Timed out while waiting for systemd to remove kubepods-burstable-poda3d6837195b14fb3d98dd6983a993c29.slice"
May 01 01:26:05.360 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-adf77ce5cc4396b144edf6bbedfc22d0 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1b3fba520934e462c37ba235581fdc74, retrying]
May 01 01:26:05.360 - 218s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-adf77ce5cc4396b144edf6bbedfc22d0 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1b3fba520934e462c37ba235581fdc74, retrying]

... 1 lines not shown

Apr 30 21:48:04.218 - 1s    E node/ci-op-clvm0thj-3a480-mcfjg-master-1 reason/FailedToDeleteCGroupsPath Apr 30 21:48:04.218950 ci-op-clvm0thj-3a480-mcfjg-master-1 kubenswrapper[2206]: I0430 21:48:04.218877    2206 pod_container_manager_linux.go:191] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pod6b7d7c84778d6c2d6c74300000545a94] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod6b7d7c84778d6c2d6c74300000545a94] : Timed out while waiting for systemd to remove kubepods-burstable-pod6b7d7c84778d6c2d6c74300000545a94.slice"
Apr 30 21:48:18.684 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1ef4170c7697251b18d0785a62c7efae expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a1458353ad1e9d0adb1d0b739e4d89f9, retrying]
Apr 30 21:48:18.684 - 650s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1ef4170c7697251b18d0785a62c7efae expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a1458353ad1e9d0adb1d0b739e4d89f9, retrying]

... 1 lines not shown

Apr 30 18:36:48.430 E ns/openshift-ovn-kubernetes pod/ovnkube-node-gn8lr node/ci-op-p3skvzxg-3a480-rwbns-worker-b-9bjgk uid/dc55f7df-f8f4-47d3-908f-12475c2513c2 container/ovnkube-node reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 18:36:51.296 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3f6859f1f7cd4a2c97e7f1826513b7e4 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-64b463d4f2e122814d73d55ddcdf75f1, retrying]
Apr 30 18:36:51.296 - 227s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3f6859f1f7cd4a2c97e7f1826513b7e4 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-64b463d4f2e122814d73d55ddcdf75f1, retrying]

... 1 lines not shown

May 01 10:20:36.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://34.173.31.154:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 01 10:21:04.334 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14498e12bcd0fdf1b83687c571135544 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-dc3ecb018bcd98afa8837b716355813a, retrying]
May 01 10:21:04.334 - 451s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14498e12bcd0fdf1b83687c571135544 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-dc3ecb018bcd98afa8837b716355813a, retrying]

... 1 lines not shown

May 01 08:31:09.199 E ns/e2e-k8s-sig-apps-job-upgrade-5888 pod/foo-qnj8n node/ip-10-0-153-37.ec2.internal uid/b7118500-68be-4a39-b8e7-85bf0add5a70 container/c reason/ContainerExit code/137 cause/Error
May 01 08:32:28.095 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-01-064657: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-59acfe2ef87c08c8ec52f52316a7f88b expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-764e4330064601e86aabf784cd373de5, retrying]
May 01 08:32:28.095 - 554s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-01-064657: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-59acfe2ef87c08c8ec52f52316a7f88b expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-764e4330064601e86aabf784cd373de5, retrying]

... 1 lines not shown

Apr 30 23:58:21.175 E ns/e2e-k8s-sig-apps-job-upgrade-4599 pod/foo-dl26j node/ip-10-0-213-147.us-west-2.compute.internal uid/861a90f9-7c6c-443a-b389-a112be0936d0 container/c reason/ContainerExit code/137 cause/Error
Apr 30 23:59:23.613 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-220027: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b14e5491877b0d5111b6b95914d96610 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b2ca20818a2ab46f709f78ad8613d63c, retrying]
Apr 30 23:59:23.613 - 595s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-220027: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b14e5491877b0d5111b6b95914d96610 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b2ca20818a2ab46f709f78ad8613d63c, retrying]

... 1 lines not shown

Apr 30 19:44:13.612 E ns/openshift-multus pod/multus-admission-controller-b584c9fb4-c9phl node/ip-10-0-222-148.us-west-2.compute.internal uid/04bad73f-f9c3-457b-aeb7-a23601e03fd7 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
Apr 30 19:47:18.567 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9539135bc6b6cd908bc44382a7d36bd3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1c37907f676d5ca601f4723e07e8db87, retrying]
Apr 30 19:47:18.567 - 175s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9539135bc6b6cd908bc44382a7d36bd3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1c37907f676d5ca601f4723e07e8db87, retrying]

... 1 lines not shown

Apr 30 16:31:06.580 E ns/openshift-cloud-credential-operator pod/pod-identity-webhook-779f79498c-729qp node/ip-10-0-139-91.us-west-1.compute.internal uid/a5f75200-55ee-4a7e-88e5-b8110062f450 container/pod-identity-webhook reason/ContainerExit code/137 cause/Error
Apr 30 16:33:48.849 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7d7514d4882e55fc869d2bad3b015480 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-52e5f798a107d039e2c9eb8bee3c9fbf, retrying]
Apr 30 16:33:48.849 - 183s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7d7514d4882e55fc869d2bad3b015480 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-52e5f798a107d039e2c9eb8bee3c9fbf, retrying]

... 1 lines not shown

May 01 08:23:05.216 - 123s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 01 08:23:14.427 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-01-064657: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-30b60a7cb4067d2c9ec672b8931cf648 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-63efcc234d252867a5b6ce388e27540c, retrying]
May 01 08:23:14.427 - 540s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-01-064657: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-30b60a7cb4067d2c9ec672b8931cf648 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-63efcc234d252867a5b6ce388e27540c, retrying]

... 1 lines not shown

Apr 30 23:40:23.703 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-797fs node/ip-10-0-219-20.us-east-2.compute.internal uid/cc681c43-6e26-4496-9e27-8ee9779c739a container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
Apr 30 23:43:36.673 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-220027: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-26eaca5ff8e1ae172f3bda7628756f5e expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-84b687c31096af5c564d4fcf0ebcce42, retrying]
Apr 30 23:43:36.673 - 143s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-220027: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-26eaca5ff8e1ae172f3bda7628756f5e expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-84b687c31096af5c564d4fcf0ebcce42, retrying]

... 1 lines not shown

Apr 30 16:19:59.031 - 114s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
Apr 30 16:20:04.403 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cc5f477d7bb02c45cd10c22d07d4afc3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e14b62fe44c49e9303466ded3c1dc1ff, retrying]
Apr 30 16:20:04.403 - 148s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-30-142901: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cc5f477d7bb02c45cd10c22d07d4afc3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e14b62fe44c49e9303466ded3c1dc1ff, retrying]

... 1 lines not shown

May 01 07:32:20.395 E ns/openshift-kni-infra pod/haproxy-host4.cluster15.ocpci.eng.rdu2.redhat.com node/host4.cluster15.ocpci.eng.rdu2.redhat.com uid/528fa206-9b22-48fb-b43b-ae80164ceae8 container/verify-api-int-resolvable reason/ContainerExit code/6 cause/Error
May 01 07:34:27.114 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c2096d8307c5358ff5cda56f9fddbd46 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-604be076b3e0acd99f4725ade1462070, retrying]
May 01 07:34:27.114 - 131s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c2096d8307c5358ff5cda56f9fddbd46 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-604be076b3e0acd99f4725ade1462070, retrying]

... 1 lines not shown

May 01 01:28:12.607 E ns/openshift-machine-api pod/metal3-5bfccc5fb7-7bcxv node/host4.cluster15.ocpci.eng.rdu2.redhat.com uid/48183ec1-da93-4542-b28a-add5a46c0489 container/metal3-static-ip-set reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 01:29:42.140 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ce519d2e95e4d05909ae68429af9a354 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6007e103b3abc9e228c7cd50866f84bb, retrying]
May 01 01:29:42.140 - 523s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ce519d2e95e4d05909ae68429af9a354 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6007e103b3abc9e228c7cd50866f84bb, retrying]

... 1 lines not shown

Apr 30 18:55:28.714 - 54ms  E clusteroperator/dns condition/Degraded status/True reason/DNS default is degraded
Apr 30 18:56:41.177 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e0e4d18b0bfc8b6cd5ad43d0c7df7e83 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d77210a0e051816d2e798cf39692458c, retrying]
Apr 30 18:56:41.177 - 510s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e0e4d18b0bfc8b6cd5ad43d0c7df7e83 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d77210a0e051816d2e798cf39692458c, retrying]

... 1 lines not shown

May 01 07:01:07.249 E ns/openshift-cluster-csi-drivers pod/gcp-pd-csi-driver-node-mggtd node/ci-op-3kp3scw0-4ef92-r5hsj-worker-b-jlhq4 uid/30151596-9561-4594-88fa-4bab1a4ea6ee container/csi-driver reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 07:01:12.364 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-353954664bc737c6fecd922f19e84ae6 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a3f41aef2d00be83b7fb3b784d192f1c, retrying]
May 01 07:01:12.364 - 610s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-353954664bc737c6fecd922f19e84ae6 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a3f41aef2d00be83b7fb3b784d192f1c, retrying]

... 1 lines not shown

May 01 01:01:22.906 E ns/openshift-e2e-loki pod/loki-promtail-hdftv node/ci-op-ivf913yp-4ef92-4gmtv-worker-c-d8z4w uid/7d4b7c9f-7b3e-46f3-8fa4-d0d0f64e2722 container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 01:03:03.304 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c376f297878fe0422e625e4b6144611d expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-770a37af1c84681ea7bd9d5f0bedde8e, retrying]
May 01 01:03:03.304 - 203s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c376f297878fe0422e625e4b6144611d expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-770a37af1c84681ea7bd9d5f0bedde8e, retrying]

... 1 lines not shown

Apr 30 18:13:30.285 E ns/openshift-network-diagnostics pod/network-check-target-8k55q node/ci-op-kmtdngpk-4ef92-z69dg-worker-b-d65rp uid/f7a34570-d28a-48e4-8939-9ef28e318fe9 container/network-check-target-container reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 18:13:31.224 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c49f981287b08bab7a7c9d52282ab673 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-53fb4664c622af5d96aa2c19338941ae, retrying]
Apr 30 18:13:31.224 - 671s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c49f981287b08bab7a7c9d52282ab673 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-53fb4664c622af5d96aa2c19338941ae, retrying]

... 1 lines not shown

May 01 07:00:12.000 - 1s    E ns/openshift-image-registry route/test-disruption-reused disruption/image-registry connection/reused reason/DisruptionBegan ns/openshift-image-registry route/test-disruption-reused disruption/image-registry connection/reused stopped responding to GET requests over reused connections: Get "https://test-disruption-reused-openshift-image-registry.apps.ci-op-ktrkk4pd-4f6ab.XXXXXXXXXXXXXXXXXXXXXX/healthz": net/http: timeout awaiting response headers
May 01 07:01:28.181 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f57e33e0fa7052beb32992508bea9852 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4dafad70573973c4b32bc91af6419a98, retrying]
May 01 07:01:28.181 - 82s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f57e33e0fa7052beb32992508bea9852 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4dafad70573973c4b32bc91af6419a98, retrying]

... 1 lines not shown

May 01 07:58:52.203 E ns/openshift-dns pod/dns-default-h57l5 node/worker-1.ostest.test.metalkube.org uid/d98c3634-0c71-41bf-9f27-62573f07b71e container/dns reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 08:00:09.217 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6d788aa33f7562564bdb8f00c5ce6dd6 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-135977c2743a2fdd5953dfe35191922f, retrying]
May 01 08:00:09.217 - 145s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6d788aa33f7562564bdb8f00c5ce6dd6 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-135977c2743a2fdd5953dfe35191922f, retrying]

... 1 lines not shown

May 01 02:01:40.190 E ns/openshift-dns pod/dns-default-d47ws node/worker-1.ostest.test.metalkube.org uid/b9692f6a-c260-44b3-935d-949ca98a0a64 container/dns reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 02:01:59.731 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-36c68398e3227433375c958c08a82912 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bb35c497811d7c5a68c54cfa662b09f4, retrying]
May 01 02:01:59.731 - 196s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-36c68398e3227433375c958c08a82912 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bb35c497811d7c5a68c54cfa662b09f4, retrying]

... 1 lines not shown

May 01 07:45:13.858 E ns/openshift-ovirt-infra pod/haproxy-ovirt12-2dq6q-master-0 node/ovirt12-2dq6q-master-0 uid/30975dd1-bac1-43d9-be14-657cd59bf842 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 07:47:17.512 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-25a3d948206bf2f85d97b101a59aa94a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-182656623696884957497557f47fb6ee, retrying]
May 01 07:47:17.512 - 333s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-25a3d948206bf2f85d97b101a59aa94a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-182656623696884957497557f47fb6ee, retrying]

... 1 lines not shown

May 01 06:51:45.329 E ns/openshift-multus pod/multus-additional-cni-plugins-r8699 node/ci-op-1tnbsn90-da17d-tjt2c-worker-c-vt67v uid/25e1b69d-1004-4afb-97f2-77db842399e5 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 06:53:42.922 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0aafd4e4e4765057b005877c3c8329ce expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-7f7600f7b81a47f4864099ff0333b70d, retrying]
May 01 06:53:42.922 - 216s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0aafd4e4e4765057b005877c3c8329ce expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-7f7600f7b81a47f4864099ff0333b70d, retrying]

... 1 lines not shown

May 01 00:50:21.654 E ns/openshift-console pod/console-6d45479755-78xbv node/ci-op-gm37wchx-da17d-rqq6v-master-2 uid/c5b0ca50-5d30-4b47-adb4-24ca6d57b499 container/console reason/ContainerExit code/2 cause/Error W0501 00:21:49.169124       1 main.go:229] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!\nI0501 00:21:49.169202       1 main.go:359] cookies are secure!\nI0501 00:21:49.258702       1 main.go:836] Binding to [::]:8443...\nI0501 00:21:49.259500       1 main.go:838] using TLS\nI0501 00:21:52.259840       1 metrics.go:138] serverconfig.Metrics: Update ConsolePlugin metrics...\nI0501 00:21:52.287155       1 metrics.go:148] serverconfig.Metrics: Update ConsolePlugin metrics: &map[] (took 27.171422ms)\nI0501 00:21:54.259831       1 metrics.go:88] usage.Metrics: Count console users...\nI0501 00:21:54.671896       1 metrics.go:174] usage.Metrics: Update console users metrics: 0 kubeadmin, 0 cluster-admins, 0 developers, 0 unknown/errors (took 412.019554ms)\n
May 01 00:52:59.665 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-87d86179f036929ef4e4dfe398bdc8b4 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fd72be5c8c6cfbb73ab46a5777969746, retrying]
May 01 00:52:59.665 - 186s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-87d86179f036929ef4e4dfe398bdc8b4 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fd72be5c8c6cfbb73ab46a5777969746, retrying]

... 1 lines not shown

Apr 30 17:59:53.515 E ns/openshift-cluster-csi-drivers pod/gcp-pd-csi-driver-node-6jzbj node/ci-op-64jlkizn-da17d-vsnvw-worker-b-m49mg uid/b7b98b04-a86e-4e9d-badb-fac6eeef1bca container/csi-node-driver-registrar reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 17:59:53.875 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-12cfef8916b81f92f0676fca413d96d4 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-8b88468e4acd7b12883a051a9b936c78, retrying]
Apr 30 17:59:53.875 - 662s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-12cfef8916b81f92f0676fca413d96d4 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-8b88468e4acd7b12883a051a9b936c78, retrying]

... 1 lines not shown

May 01 06:36:51.709 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-0fghq9l9-7f65d-s47rc-worker-eastus23-j7nn6 uid/525db111-64fd-4a49-ad0a-61e826e70965 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-01T06:12:02.358760733Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-05-01T06:12:02.358846139Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-01T06:12:02.359123159Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=error ts=2024-05-01T06:12:02.362980731Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9093/-/reload\": dial tcp [::1]:9093: connect: connection refused"\nlevel=info ts=2024-05-01T06:12:07.373666909Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/alertmanager/config/alertmanager.yaml.gz cfg_out=/etc/alertmanager/config_out/alertmanager.env.yaml watched_dirs="/etc/alertmanager/secrets/alertmanager-main-tls, /etc/alertmanager/secrets/alertmanager-main-proxy, /etc/alertmanager/secrets/alertmanager-kube-rbac-proxy, /etc/alertmanager/secrets/alertmanager-kube-rbac-proxy-metric"\nlevel=info ts=2024-05-01T06:12:07.37381492Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/alertmanager/config/alertmanager.yaml.gz out=/etc/alertmanager/config_out/alertmanager.env.yaml dirs=/etc/alertmanager/secrets/alertmanager-main-tls,/etc/alertmanager/secrets/alertmanager-main-proxy,/etc/alertmanager/secrets/alertmanager-kube-rbac-proxy,/etc/alertmanager/secrets/alertmanager-kube-rbac-proxy-metric\n
May 01 06:38:02.117 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-01-031456: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c847f0f1fb139742642da7abb2cc630e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b9a3d073d2663886d2b1806b3d351221, retrying]
May 01 06:38:02.117 - 413s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-01-031456: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c847f0f1fb139742642da7abb2cc630e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b9a3d073d2663886d2b1806b3d351221, retrying]

... 1 lines not shown

Apr 30 22:47:06.644 E ns/openshift-machine-config-operator pod/machine-config-server-9jl6r node/ci-op-bpsvcl0x-7f65d-mvd4n-master-2 uid/5a9834b3-2533-4c24-a21a-b4cea928cc91 container/machine-config-server reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
Apr 30 22:47:09.164 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b2665d2aae44c276f83b12ce845da32a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-410d7f6d2f2d55769a221a6accd927a9, retrying]
Apr 30 22:47:09.164 - 35s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b2665d2aae44c276f83b12ce845da32a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-410d7f6d2f2d55769a221a6accd927a9, retrying]

... 1 lines not shown

Apr 30 13:14:19.566 E ns/e2e-k8s-sig-apps-job-upgrade-2841 pod/foo-c4tgr node/ci-op-q6xt0864-7f65d-rtwgh-worker-eastus23-8lbw9 uid/8feb5f21-7174-44f2-be8b-fd6bda4cbdfe container/c reason/ContainerExit code/137 cause/Error
Apr 30 13:14:29.462 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4b3361a728f063a5f58143c47561b2d0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e6a40dd1588b2d24af00c7781384b641, retrying]
Apr 30 13:14:29.462 - 410s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-113036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4b3361a728f063a5f58143c47561b2d0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e6a40dd1588b2d24af00c7781384b641, retrying]

... 1 lines not shown

May 01 06:52:12.055 E ns/openshift-multus pod/multus-admission-controller-558db7cf67-x7qlk node/ip-10-0-142-220.us-west-2.compute.internal uid/ece20215-458e-455d-883b-8e3d93fe5f74 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 01 06:54:50.966 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7d4a8abe611841a799ba0b0a3df7dd58 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9cf9cb1b8b7b1a659d950224c5124c74, retrying]
May 01 06:54:50.966 - 227s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-01-050821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7d4a8abe611841a799ba0b0a3df7dd58 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9cf9cb1b8b7b1a659d950224c5124c74, retrying]

... 1 lines not shown

May 01 00:54:49.460 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-ss22v node/ip-10-0-134-248.us-west-1.compute.internal uid/b9d9699a-5139-4afe-922a-0262f187f331 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 01 00:57:26.519 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9b546b4f520bbedb8f36dbfdab823063 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ab1399b8678a3697dcbc28e4372c1777, retrying]
May 01 00:57:26.519 - 204s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-230821: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9b546b4f520bbedb8f36dbfdab823063 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ab1399b8678a3697dcbc28e4372c1777, retrying]

... 1 lines not shown

Apr 30 18:04:46.685 E ns/openshift-cloud-credential-operator pod/pod-identity-webhook-744bf47477-kdccs node/ip-10-0-209-76.us-east-2.compute.internal uid/094b03b6-6364-483e-b5ee-3cdf11b0fc72 container/pod-identity-webhook reason/ContainerExit code/137 cause/Error
Apr 30 18:07:54.980 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8180982c686d613ffe6839e128d10e12 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bb3faa4c3cb18600e5891d705ead1189, retrying]
Apr 30 18:07:54.980 - 155s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-30-162036: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8180982c686d613ffe6839e128d10e12 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bb3faa4c3cb18600e5891d705ead1189, retrying]

... 1 lines not shown

May 01 06:47:41.574 E ns/openshift-multus pod/multus-additional-cni-plugins-tj77l node/ip-10-0-251-125.ec2.internal uid/1387ab6a-c70c-427a-bd83-c358f32211a1 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 06:49:44.974 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2ba23842bef9d2325ea142fd1b5dab55 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ea26d2cfcb4b0456992fc53c1f0739bf, retrying]
May 01 06:49:44.974 - 474s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2ba23842bef9d2325ea142fd1b5dab55 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ea26d2cfcb4b0456992fc53c1f0739bf, retrying]

... 1 lines not shown

May 01 04:54:54.027 E ns/openshift-multus pod/multus-admission-controller-85d6845b6b-7rvgm node/ip-10-0-229-165.us-west-2.compute.internal uid/6dd6398a-60c8-4efc-bafb-89cbf94aa133 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 01 04:57:13.435 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-01-030927: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6a1ce21f4a1571c557d78c5526f928f3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5f96f49f74c88f5a6e5dbe6d59ec4f97, retrying]
May 01 04:57:13.435 - 243s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-01-030927: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6a1ce21f4a1571c557d78c5526f928f3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5f96f49f74c88f5a6e5dbe6d59ec4f97, retrying]

... 1 lines not shown

Apr 30 23:35:47.856 - 120s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
Apr 30 23:35:52.663 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-04-30-215955: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d85ca1fe0edc8d031fd133d79c6307e6 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-0f27f51eb1a6230ad005f5103dbdb0ec, retrying]
Apr 30 23:35:52.663 - 524s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-04-30-215955: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d85ca1fe0edc8d031fd133d79c6307e6 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-0f27f51eb1a6230ad005f5103dbdb0ec, retrying]

... 1 lines not shown

Apr 30 20:53:57.696 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-r26dk node/ip-10-0-168-23.us-west-2.compute.internal uid/69c9ed91-f58d-493b-b836-e14398cb9652 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
Apr 30 20:56:15.011 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-04-30-190353: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9e7f8e218048202b052d00b1f7f281e7 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6411a986c4b2445122a753f7a8b1643c, retrying]
Apr 30 20:56:15.011 - 224s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-04-30-190353: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9e7f8e218048202b052d00b1f7f281e7 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6411a986c4b2445122a753f7a8b1643c, retrying]

... 1 lines not shown

Apr 30 17:56:59.206 E ns/e2e-k8s-sig-apps-job-upgrade-9187 pod/foo-p65fj node/ip-10-0-228-76.us-east-2.compute.internal uid/84b03153-c9f8-4760-ac0b-4de2582e3562 container/c reason/ContainerExit code/137 cause/Error
Apr 30 17:58:36.286 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-04-30-162037: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8665473681a74fa13cd6cb5efaa1d81f expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-17bd651ecd018a438e96c7eb0b506e86, retrying]
Apr 30 17:58:36.286 - 575s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-04-30-162037: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8665473681a74fa13cd6cb5efaa1d81f expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-17bd651ecd018a438e96c7eb0b506e86, retrying]

... 1 lines not shown

May 01 05:06:58.052 E ns/e2e-k8s-sig-apps-job-upgrade-4368 pod/foo-9xvp2 node/ip-10-0-233-195.us-east-2.compute.internal uid/74de3939-8dbf-48d6-a4b6-2175fc3c0ea1 container/c reason/ContainerExit code/137 cause/Error
May 01 05:08:16.408 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-01-031000: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fb61dc21a114f4b747d70124103e0be1 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b2674097d3fdae374b466b5ba91a9ff9, retrying]
May 01 05:08:16.408 - 572s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-01-031000: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fb61dc21a114f4b747d70124103e0be1 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b2674097d3fdae374b466b5ba91a9ff9, retrying]

... 1 lines not shown

Apr 30 21:30:31.945 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-187-39.us-east-2.compute.internal uid/6ea8140d-d5ac-40fc-8496-5d8e514d1002 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-04-30T21:30:28.039Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=c5f095c646ea4c1f9524833e8e3579b827a501aa)"\nts=2024-04-30T21:30:28.039Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@929f9e3118d2, date=20240425-19:10:00)"\nts=2024-04-30T21:30:28.115Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-04-30T21:30:28.150Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-04-30T21:30:28.150Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-04-30T21:30:28.150Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=35.066181ms\n
Apr 30 21:30:43.922 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-04-30-192610: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-20229ac06ea6af991d63c7a371b8e3b3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-d811a2a3bdca69c7d799774562b68bbf, retrying]
Apr 30 21:30:43.922 - 148s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-04-30-192610: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-20229ac06ea6af991d63c7a371b8e3b3 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-d811a2a3bdca69c7d799774562b68bbf, retrying]

... 1 lines not shown

May 01 09:46:50.742 E ns/e2e-k8s-sig-apps-daemonset-upgrade-9760 pod/ds1-6gbg6 node/libvirt-s390x-2-1-f88-plr2w-worker-0-t2477 uid/b03cb8a7-b1e0-4781-afbd-fc3acf5b8ca6 container/app reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 01 09:46:57.546 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-01-030926: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3f81c247332500072f4aad93ff358ce8 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5aa0a219e730f1bea4d745c4fa3df819, retrying]
May 01 09:46:57.546 - 632s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-01-030926: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3f81c247332500072f4aad93ff358ce8 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5aa0a219e730f1bea4d745c4fa3df819, retrying]

... 1 lines not shown

Apr 30 22:12:38.900 E ns/openshift-kube-controller-manager pod/kube-controller-manager-libvirt-s390x-4-0-f88-4cvr6-master-0 node/libvirt-s390x-4-0-f88-4cvr6-master-0 uid/7786f676-3c63-41dc-988a-a2684c26f6a9 container/kube-controller-manager reason/ContainerExit code/1 cause/Error " as it is more than 1h0m0s old and approved.\nI0430 22:11:21.708709       1 cleaner.go:172] Cleaning CSR "csr-lmrlm" as it is more than 1h0m0s old and approved.\nI0430 22:11:25.919907       1 job_controller.go:514] enqueueing job e2e-k8s-sig-apps-job-upgrade-8317/foo\nI0430 22:11:26.055976       1 job_controller.go:514] enqueueing job e2e-k8s-sig-apps-job-upgrade-8317/foo\nI0430 22:11:26.108362       1 job_controller.go:514] enqueueing job e2e-k8s-sig-apps-job-upgrade-8317/foo\nI0430 22:11:26.276739       1 job_controller.go:514] enqueueing job e2e-k8s-sig-apps-job-upgrade-8317/foo\nI0430 22:11:26.404649       1 job_controller.go:514] enqueueing job e2e-k8s-sig-apps-job-upgrade-8317/foo\nI0430 22:11:26.438605       1 job_controller.go:514] enqueueing job e2e-k8s-sig-apps-job-upgrade-8317/foo\nI0430 22:11:28.554421       1 event.go:294] "Event occurred" object="openshift-monitoring/alertmanager-main" fieldPath="" kind="StatefulSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="create Pod alertmanager-main-0 in StatefulSet alertmanager-main successful"\nW0430 22:12:27.183556       1 transport.go:301] Unable to cancel request for *app.rejectIfNotReadyHeaderRT\nE0430 22:12:27.185015       1 leaderelection.go:330] error retrieving resource lock kube-system/kube-controller-manager: Get "https://api-int.libvirt-s390x-4-0-f885a.libvirt-s390x-4-0.ci:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=6s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)\nE0430 22:12:33.197730       1 leaderelection.go:367] Failed to update lock: Put "https://api-int.libvirt-s390x-4-0-f885a.libvirt-s390x-4-0.ci:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager?timeout=6s": context deadline exceeded\nI0430 22:12:33.197846       1 leaderelection.go:283] failed to renew lease kube-system/kube-controller-manager: timed out waiting for the condition\nE0430 22:12:33.197977       1 controllermanager.go:328] "leaderelection lost"\n
Apr 30 22:13:53.227 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-04-30-192536: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-61b5365b5e60f9aa04137e36422fd0de expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-e02cd43c13f59542e7a3bbfa4c37b091, retrying]
Apr 30 22:13:53.227 - 1991s E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-04-30-192536: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-61b5365b5e60f9aa04137e36422fd0de expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-e02cd43c13f59542e7a3bbfa4c37b091, retrying]

... 1 lines not shown

Apr 30 09:56:14.779 - 1s    E node/libvirt-s390x-5-1-f88-w5zbc-master-1 reason/FailedToDeleteCGroupsPath Apr 30 09:56:14.779716 libvirt-s390x-5-1-f88-w5zbc-master-1 kubenswrapper[2252]: I0430 09:56:14.779628    2252 pod_container_manager_linux.go:191] "Failed to delete cgroup paths" cgroupName=[kubepods burstable podadbe7f1c-7597-4404-95a0-009cb67d3f39] err="unable to destroy cgroup paths for cgroup [kubepods burstable podadbe7f1c-7597-4404-95a0-009cb67d3f39] : Timed out while waiting for systemd to remove kubepods-burstable-podadbe7f1c_7597_4404_95a0_009cb67d3f39.slice"
Apr 30 09:58:51.951 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-04-29-100621: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-93d74787fb16bc6434889f6e3aded7e8 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-436728a5e7de136baa3387851d521a64, retrying]
Apr 30 09:58:51.951 - 155s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-04-29-100621: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-93d74787fb16bc6434889f6e3aded7e8 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-436728a5e7de136baa3387851d521a64, retrying]

... 1 lines not shown

May 01 02:45:30.547 E ns/openshift-multus pod/multus-admission-controller-7b6b69f4c5-64wrc node/ip-10-0-143-132.us-west-2.compute.internal uid/2e871a15-2297-4681-a8d9-39b4a8ee4218 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 01 02:48:41.036 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-906d6973f58026d00559a01882f58a11 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-cae20458b58ba94cf5a7794519971b14, retrying]
May 01 02:48:41.036 - 128s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-04-30-210211: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-906d6973f58026d00559a01882f58a11 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-cae20458b58ba94cf5a7794519971b14, retrying]

... 1 lines not shown

Apr 30 23:35:41.606 - 140s  E clusteroperator/dns condition/Degraded status/True reason/DNS default is degraded
Apr 30 23:35:43.483 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.11.0-0.ci-2024-04-30-215042: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2ad13870962868f35a50a79f84461ded expected 15d0b0288a4330b90ac89f14c781dfa7349af52c has 59a8d308bfa4cd89abdec9d345e396fe9dafd7b0: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-bf8415ecb2c9189abe682a1a964c58bc, retrying]
Apr 30 23:35:43.483 - 525s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.11.0-0.ci-2024-04-30-215042: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2ad13870962868f35a50a79f84461ded expected 15d0b0288a4330b90ac89f14c781dfa7349af52c has 59a8d308bfa4cd89abdec9d345e396fe9dafd7b0: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-bf8415ecb2c9189abe682a1a964c58bc, retrying]

... 1 lines not shown

Apr 30 23:31:21.921 E ns/openshift-console pod/console-87b654478-cnfrh node/ip-10-0-153-232.us-west-1.compute.internal uid/c21c9347-4b32-416d-af92-44cfa47d8449 container/console reason/ContainerExit code/2 cause/Error W0430 23:17:08.591139       1 main.go:220] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!\nI0430 23:17:08.591161       1 main.go:364] cookies are secure!\nI0430 23:17:08.622246       1 main.go:798] Binding to [::]:8443...\nI0430 23:17:08.622276       1 main.go:800] using TLS\n
Apr 30 23:34:24.064 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.11.0-0.ci-2024-04-30-215042: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bed90dbc65b12b94de7b67cde32a6cf2 expected 15d0b0288a4330b90ac89f14c781dfa7349af52c has 59a8d308bfa4cd89abdec9d345e396fe9dafd7b0: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9a332611e0fa7e43c7bf888c61a269bb, retrying]
Apr 30 23:34:24.064 - 157s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.11.0-0.ci-2024-04-30-215042: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bed90dbc65b12b94de7b67cde32a6cf2 expected 15d0b0288a4330b90ac89f14c781dfa7349af52c has 59a8d308bfa4cd89abdec9d345e396fe9dafd7b0: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9a332611e0fa7e43c7bf888c61a269bb, retrying]

... 1 lines not shown

Apr 30 23:29:14.561 E ns/openshift-network-diagnostics pod/network-check-target-zswh9 node/ip-10-0-172-47.ec2.internal uid/6f71d3a3-09d2-4d5b-bfee-182fdd15db8d container/network-check-target-container reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
Apr 30 23:31:52.050 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.11.0-0.ci-2024-04-30-215042: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8c2e874f7157981d9baec7ceb5d04b18 expected 15d0b0288a4330b90ac89f14c781dfa7349af52c has 59a8d308bfa4cd89abdec9d345e396fe9dafd7b0: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0dddc0314f3ea24b47b590b31cd8163b, retrying]
Apr 30 23:31:52.050 - 195s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.11.0-0.ci-2024-04-30-215042: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8c2e874f7157981d9baec7ceb5d04b18 expected 15d0b0288a4330b90ac89f14c781dfa7349af52c has 59a8d308bfa4cd89abdec9d345e396fe9dafd7b0: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0dddc0314f3ea24b47b590b31cd8163b, retrying]

... 1 lines not shown

Apr 30 16:16:42.551 E ns/openshift-multus pod/multus-admission-controller-9c48d875d-v8jvp node/ci-op-0l08p6xi-efead-v4dx9-master-0 uid/1bd48ca1-4ceb-4989-b034-9224c6e0ad05 container/kube-rbac-proxy reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
Apr 30 16:18:26.754 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-29-060310: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-13f582563f4d724a13955c87a54ba699 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-371af337a12ce649721b4ad3d3e16028, retrying]
Apr 30 16:18:26.754 - 280s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-29-060310: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-13f582563f4d724a13955c87a54ba699 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-371af337a12ce649721b4ad3d3e16028, retrying]

... 1 lines not shown

Apr 30 16:15:12.991 E ns/openshift-multus pod/multus-admission-controller-558db7cf67-dp67w node/ip-10-0-239-159.us-west-2.compute.internal uid/0e3f8633-7729-4716-94c3-60f249223a3c container/multus-admission-controller reason/ContainerExit code/137 cause/Error
Apr 30 16:17:14.468 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-04-29-092236: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c0ecfc7efae77e74d41d00daa765a7bb expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-08c08aea5542a5c3b4aff2cb24a1f48a, retrying]
Apr 30 16:17:14.468 - 265s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-04-29-092236: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c0ecfc7efae77e74d41d00daa765a7bb expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-08c08aea5542a5c3b4aff2cb24a1f48a, retrying]

... 1 lines not shown

Apr 30 14:54:13.174 E ns/e2e-k8s-sig-apps-job-upgrade-8600 pod/foo-5bpmn node/ip-10-0-129-7.ec2.internal uid/24f6f2b5-ba6c-44d0-9467-1e344502d278 container/c reason/ContainerExit code/137 cause/Error
Apr 30 14:55:16.903 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.56: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ffee10daf84f1184b7c292f9ca7b1a32 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-78b776bff41c47805176aeced2d3cca4, retrying]
Apr 30 14:55:16.903 - 519s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.56: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ffee10daf84f1184b7c292f9ca7b1a32 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-78b776bff41c47805176aeced2d3cca4, retrying]

... 1 lines not shown

Apr 30 15:24:24.477 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-c84xr node/ip-10-0-232-115.us-west-1.compute.internal uid/3d9518b3-846b-4e03-ab88-383c5ab282cd container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
Apr 30 15:26:00.755 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-29-060310: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d8c50903024954ef6053265736ccca43 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1c43df03980d8b0d839bf5f7136c9049, retrying]
Apr 30 15:26:00.755 - 580s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-29-060310: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d8c50903024954ef6053265736ccca43 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1c43df03980d8b0d839bf5f7136c9049, retrying]

... 1 lines not shown

Apr 30 15:22:56.338 E ns/openshift-console pod/console-68b7bb7dc7-5dsx2 node/ip-10-0-157-150.us-west-2.compute.internal uid/1a545aa7-77df-4374-be91-b7eefa67ae83 container/console reason/ContainerExit code/2 cause/Error W0430 14:40:45.163390       1 main.go:229] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!\nI0430 14:40:45.163436       1 main.go:359] cookies are secure!\nI0430 14:40:45.217791       1 main.go:836] Binding to [::]:8443...\nI0430 14:40:45.217853       1 main.go:838] using TLS\nI0430 14:40:48.217966       1 metrics.go:138] serverconfig.Metrics: Update ConsolePlugin metrics...\nI0430 14:40:48.238232       1 metrics.go:148] serverconfig.Metrics: Update ConsolePlugin metrics: &map[] (took 20.238957ms)\nI0430 14:40:50.218713       1 metrics.go:88] usage.Metrics: Count console users...\nI0430 14:40:50.623087       1 metrics.go:174] usage.Metrics: Update console users metrics: 0 kubeadmin, 0 cluster-admins, 0 developers, 0 unknown/errors (took 404.338223ms)\n
Apr 30 15:25:26.954 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-04-29-060310: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8d6bf19de97c2c4202015a1823a39eb5 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9702e631319fa4a66997d6851ec4e985, retrying]
Apr 30 15:25:26.954 - 184s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-04-29-060310: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8d6bf19de97c2c4202015a1823a39eb5 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9702e631319fa4a66997d6851ec4e985, retrying]

... 1 lines not shown