May 16 19:09:59.286 E ns/e2e-k8s-sig-apps-job-upgrade-2698 pod/foo-d4fjs node/ip-10-0-242-9.us-west-1.compute.internal uid/61843fab-aa14-427b-8f37-ef281813fdfb container/c reason/ContainerExit code/137 cause/Error
May 16 19:10:50.342 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-16-170112: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-13d3211a3d2a11ca57930643da22e2ad expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-34757610937dfe1c50531976b3fd5bf6, retrying]
May 16 19:10:50.342 - 598s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-16-170112: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-13d3211a3d2a11ca57930643da22e2ad expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-34757610937dfe1c50531976b3fd5bf6, retrying]

... 1 lines not shown

May 15 15:32:59.679 E ns/e2e-k8s-sig-apps-job-upgrade-5241 pod/foo-gjmqv node/ip-10-0-227-40.us-east-2.compute.internal uid/d2039710-a82d-4f43-a131-a061771b2632 container/c reason/ContainerExit code/137 cause/Error
May 15 15:33:36.338 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-15-132734: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14dd2743669d225a37c46cbd84007938 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-902e8a05bafed2f4fa429d8e09ccb0ff, retrying]
May 15 15:33:36.338 - 560s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-15-132734: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14dd2743669d225a37c46cbd84007938 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-902e8a05bafed2f4fa429d8e09ccb0ff, retrying]

... 1 lines not shown

May 16 05:29:44.787 E ns/openshift-multus pod/multus-admission-controller-79f9dffc9d-88snz node/ip-10-0-224-68.ec2.internal uid/b64d06d3-027c-49e0-82b0-92846920b247 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 16 05:32:51.416 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-16-033540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1d0904667605acdd9e2049c311123e8a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-cb45ff937ae1b6d24354452ffd1a0f0e, retrying]
May 16 05:32:51.416 - 152s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-16-033540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1d0904667605acdd9e2049c311123e8a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-cb45ff937ae1b6d24354452ffd1a0f0e, retrying]

... 1 lines not shown

May 16 12:01:15.899 - 101s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 16 12:01:19.357 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-16-093540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-66726d96efdb442a029352584a0d1ee9 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ae8035744257f3afbbafbe2e3000d344, retrying]
May 16 12:01:19.357 - 137s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-16-093540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-66726d96efdb442a029352584a0d1ee9 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ae8035744257f3afbbafbe2e3000d344, retrying]

... 1 lines not shown

May 15 06:04:15.675 E ns/e2e-k8s-sig-apps-job-upgrade-3364 pod/foo-l8dz7 node/ip-10-0-157-136.ec2.internal uid/03a5e86d-9ff7-4396-9480-13748e1c8ed8 container/c reason/ContainerExit code/137 cause/Error
May 15 06:05:54.080 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-15-034503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0975ba3c364da85fda3a0ee2b560d2a5 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-da4f01176de651167131edff58cb127c, retrying]
May 15 06:05:54.080 - 525s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-15-034503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0975ba3c364da85fda3a0ee2b560d2a5 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-da4f01176de651167131edff58cb127c, retrying]

... 1 lines not shown

May 14 23:30:49.073 E ns/e2e-k8s-sig-apps-job-upgrade-6162 pod/foo-qbpqq node/ip-10-0-182-143.us-east-2.compute.internal uid/e4992e26-ebd4-447b-8450-7f723aaa5f87 container/c reason/ContainerExit code/137 cause/Error
May 14 23:32:14.684 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-14-214503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-35d944ccdd1a94c1e62c17a244a3ea56 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f2b4b925f9d5fe793f69ca6bf5b394d9, retrying]
May 14 23:32:14.684 - 565s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-14-214503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-35d944ccdd1a94c1e62c17a244a3ea56 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f2b4b925f9d5fe793f69ca6bf5b394d9, retrying]

... 1 lines not shown

May 15 09:19:22.721 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-fwzh8 node/ip-10-0-165-26.us-east-2.compute.internal uid/afa143ee-81ce-49f9-ac3f-79f266179edb container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 15 09:22:02.644 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-15-034503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5d9347cf4fcee346ee94cc1820eca381 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-af1afa25d88292bb09400fc7015469c3, retrying]
May 15 09:22:02.644 - 171s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-15-034503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5d9347cf4fcee346ee94cc1820eca381 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-af1afa25d88292bb09400fc7015469c3, retrying]

... 1 lines not shown

May 17 00:49:54.075 - 213ms E clusteroperator/storage condition/Degraded status/True reason/AWSEBSCSIDriverOperatorDeploymentDegraded: Get "https://172.30.0.1:443/apis/apps/v1/namespaces/openshift-cluster-csi-drivers/deployments/aws-ebs-csi-driver-operator": context canceled
May 17 00:51:02.263 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-16-230112: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-49cfafaf2fa9840911b424cabae572f8 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-47ac5a14e016e44d8ae52e9b4e6f66f3, retrying]
May 17 00:51:02.263 - 530s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-16-230112: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-49cfafaf2fa9840911b424cabae572f8 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-47ac5a14e016e44d8ae52e9b4e6f66f3, retrying]

... 1 lines not shown

May 14 17:42:18.658 E ns/e2e-k8s-sig-apps-job-upgrade-6896 pod/foo-xsk67 node/ip-10-0-174-199.ec2.internal uid/3c1a826e-dbe3-4505-8561-1d75d667d8ee container/c reason/ContainerExit code/137 cause/Error
May 14 17:44:09.064 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-14-154503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-65563c0c24b80325fc1adc71dc5deff6 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-015ad3cdb833bfb79d23dbb0a7a842b6, retrying]
May 14 17:44:09.064 - 519s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-14-154503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-65563c0c24b80325fc1adc71dc5deff6 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-015ad3cdb833bfb79d23dbb0a7a842b6, retrying]

... 1 lines not shown

May 14 00:00:51.597 E ns/openshift-cloud-credential-operator pod/pod-identity-webhook-85c489765d-tlqsh node/ip-10-0-252-4.us-east-2.compute.internal uid/5f8144ec-998b-4e59-81a3-a9c70bbd6081 container/pod-identity-webhook reason/ContainerExit code/137 cause/Error
May 14 00:04:01.966 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-13-211327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f6ef9212a4b7226922eabfd0b97804e3 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0f27799090f24f5b41b35042f203e303, retrying]
May 14 00:04:01.966 - 151s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-13-211327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f6ef9212a4b7226922eabfd0b97804e3 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0f27799090f24f5b41b35042f203e303, retrying]

... 1 lines not shown

May 14 11:34:10.353 E ns/openshift-cloud-credential-operator pod/pod-identity-webhook-84b8dd9698-56kmt node/ip-10-0-136-100.ec2.internal uid/d0dbc4b5-434e-418d-baa5-cf0d35d4a083 container/pod-identity-webhook reason/ContainerExit code/137 cause/Error
May 14 11:37:24.133 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-14-094503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2f71b1b3a8c7ed32b5674e278ac199b0 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-73dcf4477f1f60166300b2ab9ad86f4f, retrying]
May 14 11:37:24.133 - 147s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-14-094503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2f71b1b3a8c7ed32b5674e278ac199b0 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-73dcf4477f1f60166300b2ab9ad86f4f, retrying]

... 1 lines not shown

May 13 17:18:16.104 E clusterversion/version changed Failing to True: ClusterOperatorNotAvailable: Cluster operator control-plane-machine-set is not available
May 13 17:18:49.131 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-13-151327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-933396d5b284486508e2ff264ca4a579 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-df068dd45e54c59ea4024f1ee994e028, retrying]
May 13 17:18:49.131 - 117s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-13-151327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-933396d5b284486508e2ff264ca4a579 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-df068dd45e54c59ea4024f1ee994e028, retrying]

... 1 lines not shown

May 13 11:15:07.943 E ns/openshift-multus pod/multus-admission-controller-5866b6469d-jl8g7 node/ip-10-0-176-156.us-west-1.compute.internal uid/ee5d7d1b-d83b-4064-8a8a-6a3ad5e2cb12 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 13 11:18:14.711 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-13-091327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-284fbb3ff1b26941e4321f03151a3e63 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-32aaffb63c8b69a1a73b3ff5f2f8705b, retrying]
May 13 11:18:14.711 - 164s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-13-091327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-284fbb3ff1b26941e4321f03151a3e63 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-32aaffb63c8b69a1a73b3ff5f2f8705b, retrying]

... 1 lines not shown

May 10 16:48:24.546 E ns/e2e-k8s-sig-apps-job-upgrade-2804 pod/foo-9hgtp node/ip-10-0-132-37.us-west-1.compute.internal uid/627fb4d6-b2c0-4804-9c91-b7b52cbd5e9f container/c reason/ContainerExit code/137 cause/Error
May 10 16:49:54.337 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-10-145431: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d82f6973050a53525c220a6af94e0304 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-739588bede489abd356d93a391b29d14, retrying]
May 10 16:49:54.337 - 582s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-10-145431: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d82f6973050a53525c220a6af94e0304 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-739588bede489abd356d93a391b29d14, retrying]

... 1 lines not shown

May 10 13:39:45.734 - 103s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 10 13:40:02.892 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-10-082301: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-12110b9a36fe82676960a4b0c09b54c3 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a9bdf4637204b49776639f427ba6a5eb, retrying]
May 10 13:40:02.892 - 514s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-10-082301: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-12110b9a36fe82676960a4b0c09b54c3 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a9bdf4637204b49776639f427ba6a5eb, retrying]

... 1 lines not shown

May 10 10:28:36.084 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-172-48.ec2.internal uid/b4c1f5bf-62df-4850-941e-61be00778c38 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-10T10:28:33.288Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=release-4.13, revision=df2f11e2ddc206ad1924ed766b4a394d84c9236f)"\nts=2024-05-10T10:28:33.288Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@50ec6f22b504, date=20231031-09:45:13)"\nts=2024-05-10T10:28:33.308Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-10T10:28:33.336Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-10T10:28:33.336Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-10T10:28:33.336Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=27.913887ms\n
May 10 10:30:43.744 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-10-082301: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cd9e937343c6acd61c1cb02eaf0d437d expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-beee0364d5355996b9dd9e64034b33ae, retrying]
May 10 10:30:43.744 - 555s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-10-082301: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cd9e937343c6acd61c1cb02eaf0d437d expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-beee0364d5355996b9dd9e64034b33ae, retrying]

... 1 lines not shown

May 10 00:12:33.145 E ns/openshift-multus pod/multus-admission-controller-5c4694f9f6-db59x node/ip-10-0-251-222.us-west-2.compute.internal uid/0c6e3df1-6a51-415d-ad46-b086e34ebda3 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 00:15:33.216 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-221809: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-94bae8798480c98ac2b9f5979295e914 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4af9598ba2b07934cc0cba8fd38f4876, retrying]
May 10 00:15:33.216 - 177s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-221809: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-94bae8798480c98ac2b9f5979295e914 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4af9598ba2b07934cc0cba8fd38f4876, retrying]

... 1 lines not shown

May 09 15:34:24.684 E clusterversion/version changed Failing to True: ClusterOperatorNotAvailable: Cluster operator control-plane-machine-set is not available
May 09 15:34:26.882 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-130510: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dbc223a1043f0f909120a2e754e01d2f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-80c9e0437a2b7b47e7b675b116df15ec, retrying]
May 09 15:34:26.882 - 144s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-130510: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dbc223a1043f0f909120a2e754e01d2f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-80c9e0437a2b7b47e7b675b116df15ec, retrying]

... 1 lines not shown

May 09 02:45:43.781 - 101s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 09 02:45:57.284 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-004250: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-887c523d3d0844e30ec4b600e8b13fbc expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-487ae34fb9ab093cb97264bc3b89366d, retrying]
May 09 02:45:57.284 - 127s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-004250: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-887c523d3d0844e30ec4b600e8b13fbc expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-487ae34fb9ab093cb97264bc3b89366d, retrying]

... 1 lines not shown

May 08 14:38:19.329 E ns/openshift-multus pod/multus-admission-controller-6b54669bb-zdbw7 node/ip-10-0-188-178.us-west-2.compute.internal uid/b8de074a-7cc4-4d8f-8a4e-6c99ad4fdc43 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 08 14:41:17.619 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-08-123136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c56268e37a08c740dc87fc9e6bc20fd7 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-14364c69e4b23bd945e7723c52dfbb03, retrying]
May 08 14:41:17.619 - 189s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-08-123136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c56268e37a08c740dc87fc9e6bc20fd7 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-14364c69e4b23bd945e7723c52dfbb03, retrying]

... 1 lines not shown

May 08 08:31:24.771 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-hsgwr node/ip-10-0-183-204.us-west-2.compute.internal uid/883f8902-4236-48c8-a47a-26ff7ae3b918 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 08 08:33:10.524 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-08-063136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-939f7fae0fc65bcf74bebd55ad7a12f4 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3d35482a4b28a707d6ae866836c7b657, retrying]
May 08 08:33:10.524 - 238s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-08-063136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-939f7fae0fc65bcf74bebd55ad7a12f4 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3d35482a4b28a707d6ae866836c7b657, retrying]

... 1 lines not shown

May 08 02:00:05.040 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-242-168.us-west-2.compute.internal uid/6e58649e-36a2-4e79-ac96-e3680ca629e7 container/alertmanager reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 02:01:18.620 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-07-235844: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2ebfa68b98267214db3c4a8cf030b19 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c01fca846421f4ece7b1c9889df8d86b, retrying]
May 08 02:01:18.620 - 597s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-07-235844: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2ebfa68b98267214db3c4a8cf030b19 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c01fca846421f4ece7b1c9889df8d86b, retrying]

... 1 lines not shown

May 07 17:33:56.610 E ns/openshift-multus pod/multus-admission-controller-747dd4df94-hhwwx node/ip-10-0-186-217.ec2.internal uid/67247a9c-5eea-4929-93a0-1a1fd395c8d7 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 07 17:37:34.296 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-07-154117: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a800c8f858d9f91f7b731466796d29dd expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ec442d69cabfd2f6d6ca04c47050aa9f, retrying]
May 07 17:37:34.296 - 163s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-07-154117: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a800c8f858d9f91f7b731466796d29dd expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ec442d69cabfd2f6d6ca04c47050aa9f, retrying]

... 1 lines not shown

May 07 10:51:47.079 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-jr72x node/ip-10-0-172-45.us-west-2.compute.internal uid/78753a0d-83d3-4a69-80c0-d5b1bb7e8960 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 07 10:54:37.570 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-07-085425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-680f6e2e520810a0e116fef355015233 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a42c676183da517a29574eae8762d87c, retrying]
May 07 10:54:37.570 - 145s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-07-085425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-680f6e2e520810a0e116fef355015233 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a42c676183da517a29574eae8762d87c, retrying]

... 1 lines not shown

May 06 21:27:23.636 - 118s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 06 21:27:23.701 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-06-192520: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d8bc2b69b313a23dea36152482ca5a6e expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-66de88edf513b9e703017c72aa8a9cc0, retrying]
May 06 21:27:23.701 - 159s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-06-192520: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d8bc2b69b313a23dea36152482ca5a6e expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-66de88edf513b9e703017c72aa8a9cc0, retrying]

... 1 lines not shown

May 03 18:25:18.366 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-206-44.us-west-2.compute.internal uid/998560fc-2a6f-480b-8ee3-ec83d2a019fc container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-03T17:50:43.706703286Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.63.0, branch=release-4.13, revision=30fdccd13)"\nlevel=info ts=2024-05-03T17:50:43.706760577Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/amd64, user=root, date=20231108-08:14:29)"\nlevel=info ts=2024-05-03T17:50:43.70691216Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-03T17:50:47.908976163Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-03T17:50:47.909062505Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-03T17:52:10.175094896Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-03T17:53:31.18401006Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-03T17:54:53.125680832Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prom
May 03 18:26:00.279 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-03-162932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-669b8e91790c35f74a0c3a357923a1fc expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3f5b6c94a295df1f0bc718c7bf79a036, retrying]
May 03 18:26:00.279 - 631s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-03-162932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-669b8e91790c35f74a0c3a357923a1fc expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3f5b6c94a295df1f0bc718c7bf79a036, retrying]

... 1 lines not shown

May 06 14:53:45.773 E ns/openshift-cloud-credential-operator pod/pod-identity-webhook-6f9d58d7b7-bxrgl node/ip-10-0-220-196.us-west-1.compute.internal uid/3ffd5ca3-d623-4fe4-86c9-9943e8f84c6b container/pod-identity-webhook reason/ContainerExit code/137 cause/Error
May 06 14:56:50.121 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-06-124130: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5722b6d542af36adc3bb66b333b7b950 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-59ac562a625e8d66b5323754ee02dc42, retrying]
May 06 14:56:50.121 - 151s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-06-124130: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5722b6d542af36adc3bb66b333b7b950 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-59ac562a625e8d66b5323754ee02dc42, retrying]

... 1 lines not shown

May 03 12:30:39.134 E clusterversion/version changed Failing to True: ClusterOperatorNotAvailable: Cluster operator control-plane-machine-set is not available
May 03 12:31:03.878 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-03-102932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eb237839fa924e6fa9045abba80d6dec expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e3bdd377c51714430b375ea7174afde4, retrying]
May 03 12:31:03.878 - 125s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-03-102932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eb237839fa924e6fa9045abba80d6dec expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e3bdd377c51714430b375ea7174afde4, retrying]

... 1 lines not shown