May 16 21:05:59.987 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-174-158.us-west-1.compute.internal uid/dbfb0e05-c41f-4541-b442-fed8fc9ffe1a container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-16T21:05:57.094Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=cca15d1d9be3ebe61f12f5179e54f0c4c8b23e8b)"\nts=2024-05-16T21:05:57.094Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@9fa3cbc52ef3, date=20240515-05:16:11)"\nts=2024-05-16T21:05:57.132Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-16T21:05:57.169Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-16T21:05:57.169Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-16T21:05:57.169Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=36.945683ms\n
May 16 21:06:18.528 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-16-190046: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7ca7b38cdbc7c7a472940c9dd57877ae expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b50d57d02c852950574d84f37a379a5d, retrying]
May 16 21:06:18.528 - 691s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-16-190046: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7ca7b38cdbc7c7a472940c9dd57877ae expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b50d57d02c852950574d84f37a379a5d, retrying]

... 1 lines not shown

May 16 14:59:41.611 E ns/openshift-multus pod/multus-admission-controller-648cf4bf59-wh4bw node/ip-10-0-192-183.us-west-1.compute.internal uid/c2dba02a-1c14-4c4e-8a97-e11e1dd3bff2 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 16 15:01:47.098 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-16-131309: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0734b5fde224993dd192e5938e57f708 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-204a4220969b4cd13948a7d38790cc2f, retrying]
May 16 15:01:47.098 - 258s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-16-131309: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0734b5fde224993dd192e5938e57f708 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-204a4220969b4cd13948a7d38790cc2f, retrying]

... 1 lines not shown

May 15 16:28:23.417 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-176-143.us-east-2.compute.internal uid/836371b1-8c40-4860-91b7-3a457575bd5b container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-15T16:28:21.341Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=cca15d1d9be3ebe61f12f5179e54f0c4c8b23e8b)"\nts=2024-05-15T16:28:21.342Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@9fa3cbc52ef3, date=20240515-05:16:11)"\nts=2024-05-15T16:28:21.361Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-15T16:28:21.397Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-15T16:28:21.397Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-15T16:28:21.397Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=35.507961ms\n
May 15 16:30:47.845 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-15-145210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-837b31ae2556a9de34c9859001faec27 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-17db096f030d8b5ef1232542bfe0957e, retrying]
May 15 16:30:47.845 - 550s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-15-145210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-837b31ae2556a9de34c9859001faec27 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-17db096f030d8b5ef1232542bfe0957e, retrying]

... 1 lines not shown

May 16 07:02:20.969 E ns/openshift-multus pod/multus-admission-controller-648cf4bf59-25kcb node/ip-10-0-148-221.us-east-2.compute.internal uid/5c443574-d761-46eb-9e9d-e7ba82626099 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 16 07:05:12.649 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-16-052446: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1933f90e0ea5e7443cd6f58f2d547d29 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a8ab961bf6c83b2ede1194e0f9e28acc, retrying]
May 16 07:05:12.649 - 150s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-16-052446: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1933f90e0ea5e7443cd6f58f2d547d29 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a8ab961bf6c83b2ede1194e0f9e28acc, retrying]

... 1 lines not shown

May 15 12:50:04.423 E ns/openshift-oauth-apiserver pod/apiserver-5d64ccb9c8-zxwdk node/ip-10-0-206-101.us-west-2.compute.internal uid/57854564-630a-41c6-82d2-a101d691aa66 container/oauth-apiserver reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 15 12:52:11.438 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-15-105359: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-86bc352058a4bf0eb9de983a92afdc45 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-880682fb07e131c5c18d5eb35c1290ab, retrying]
May 15 12:52:11.438 - 214s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-15-105359: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-86bc352058a4bf0eb9de983a92afdc45 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-880682fb07e131c5c18d5eb35c1290ab, retrying]

... 1 lines not shown

May 11 02:18:24.892 E ns/openshift-multus pod/multus-admission-controller-6f49b7cb55-5sxqv node/ip-10-0-138-107.us-west-1.compute.internal uid/6889a712-d026-4800-926d-f35de75f0ef5 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 11 02:20:35.208 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-142436: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a2a6d050a77eade7d5121610d4628cf6 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1205ad0140431cfa5063aa87d61bc893, retrying]
May 11 02:20:35.208 - 239s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-142436: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a2a6d050a77eade7d5121610d4628cf6 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-1205ad0140431cfa5063aa87d61bc893, retrying]

... 1 lines not shown

May 14 16:43:41.671 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-155-55.us-west-2.compute.internal uid/90ff29b1-eec0-4627-a01b-1dce9e869331 container/alertmanager reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 16:43:48.224 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-14-150142: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b988cfd82ff8ad40bdc653ad7debf74f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3a9605e71d1321583e9da7cab545b894, retrying]
May 14 16:43:48.224 - 682s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-14-150142: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b988cfd82ff8ad40bdc653ad7debf74f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3a9605e71d1321583e9da7cab545b894, retrying]

... 1 lines not shown

May 14 13:08:40.722 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-165-4.us-west-2.compute.internal uid/6939e7d5-8420-4487-9bc2-0a4fdbecc9ee container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-14T13:08:38.804Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=38194c1bde7b693bb84526d1084400a1e76da31b)"\nts=2024-05-14T13:08:38.805Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@cb558022914e, date=20240510-00:07:09)"\nts=2024-05-14T13:08:38.825Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-14T13:08:38.858Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-14T13:08:38.859Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-14T13:08:38.859Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=34.10082ms\n
May 14 13:09:10.043 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-14-111035: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a647d0b25a59f6607f39aa41b931467d expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d54464fb595b4cdfc7cf84a566f7fece, retrying]
May 14 13:09:10.043 - 696s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-14-111035: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a647d0b25a59f6607f39aa41b931467d expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d54464fb595b4cdfc7cf84a566f7fece, retrying]

... 1 lines not shown

May 14 09:44:18.719 E clusterversion/version changed Failing to True: ClusterOperatorNotAvailable: Cluster operator control-plane-machine-set is not available
May 14 09:44:53.300 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-14-080344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c3e650735e6505f577a071338a055355 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2d908838feba3f47dd96a364b2754223, retrying]
May 14 09:44:53.300 - 123s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-14-080344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c3e650735e6505f577a071338a055355 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2d908838feba3f47dd96a364b2754223, retrying]

... 1 lines not shown

May 10 16:07:59.861 E ns/openshift-multus pod/multus-admission-controller-6f49b7cb55-v7v6j node/ip-10-0-187-123.us-west-2.compute.internal uid/46887859-c12a-4486-9d90-c42e2f4ba973 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 16:10:25.449 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-142436: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4cf15af590eed849f6e01b6e70d8e570 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bee064ef629bddda6c774f895b94d357, retrying]
May 10 16:10:25.449 - 219s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-142436: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4cf15af590eed849f6e01b6e70d8e570 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bee064ef629bddda6c774f895b94d357, retrying]

... 1 lines not shown

May 10 10:30:45.290 E ns/e2e-k8s-sig-apps-job-upgrade-6012 pod/foo-8jdhx node/ip-10-0-140-77.us-west-2.compute.internal uid/9ccbb280-6cb0-403d-86d2-e894ad271e5c container/c reason/ContainerExit code/137 cause/Error
May 10 10:31:33.836 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-084543: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4813db6a3ee16c99931b0c30d609f632 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-302cde450ca362bc6e4e1ec013885479, retrying]
May 10 10:31:33.836 - 672s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-084543: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4813db6a3ee16c99931b0c30d609f632 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-302cde450ca362bc6e4e1ec013885479, retrying]

... 1 lines not shown

May 10 13:13:30.794 E ns/e2e-k8s-sig-apps-job-upgrade-3477 pod/foo-npnc4 node/ip-10-0-141-21.us-east-2.compute.internal uid/8b048697-8e3f-4759-9a72-74d21452dbb8 container/c reason/ContainerExit code/137 cause/Error
May 10 13:14:53.518 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-114355: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e317ba45d7896a4b19ec8c2c7c748395 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d36ccd78e28e96c6dad0053308484a06, retrying]
May 10 13:14:53.518 - 549s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-114355: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e317ba45d7896a4b19ec8c2c7c748395 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d36ccd78e28e96c6dad0053308484a06, retrying]

... 1 lines not shown

May 10 06:41:19.330 E ns/openshift-multus pod/multus-admission-controller-6f49b7cb55-qzt57 node/ip-10-0-239-25.us-west-2.compute.internal uid/d77c324c-8db2-40c6-b2a9-5cc56a7e431b container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 06:43:50.614 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-045750: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-823147e71730467278d9f0264ed98047 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-15438745105b02a5112f86779785cec2, retrying]
May 10 06:43:50.614 - 233s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-10-045750: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-823147e71730467278d9f0264ed98047 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-15438745105b02a5112f86779785cec2, retrying]

... 1 lines not shown

May 09 14:29:06.861 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-gljww node/ip-10-0-166-18.us-west-2.compute.internal uid/e4a9a115-1cc9-41ef-9b1a-6bc36b5f7975 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 09 14:31:15.943 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-09-124017: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fd0e560b59eb7259aac47b9c6c7d6663 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-c62d6a9e063f2b73640dcd9c9ce4b304, retrying]
May 09 14:31:15.943 - 228s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-09-124017: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fd0e560b59eb7259aac47b9c6c7d6663 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-c62d6a9e063f2b73640dcd9c9ce4b304, retrying]

... 1 lines not shown

May 08 19:19:56.052 E ns/e2e-k8s-sig-apps-job-upgrade-8818 pod/foo-l6j9b node/ip-10-0-160-172.us-west-1.compute.internal uid/04ae29bd-2147-4b30-861d-22776b8d536a container/c reason/ContainerExit code/137 cause/Error
May 08 19:20:43.889 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-08-174140: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4c7a46c187f0f1b5cf8fd196c7c39fd4 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ad05f15095ba34bcc397f065d5f90479, retrying]
May 08 19:20:43.889 - 662s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-08-174140: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4c7a46c187f0f1b5cf8fd196c7c39fd4 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ad05f15095ba34bcc397f065d5f90479, retrying]

... 1 lines not shown

May 04 02:19:42.474 - 96s   E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 04 02:19:54.823 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-02-145023: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-54c75306cf5e5491d43ef087098a33ef expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2c5014ce8becd6abb15069968991f116, retrying]
May 04 02:19:54.823 - 123s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-02-145023: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-54c75306cf5e5491d43ef087098a33ef expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2c5014ce8becd6abb15069968991f116, retrying]

... 1 lines not shown

May 08 13:39:35.615 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-231-104.us-west-1.compute.internal uid/30fdc879-0374-4757-a636-2041329fe273 container/alertmanager reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 13:39:59.848 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-08-115725: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e1eedec6b04495b7e6f218adec22341f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-45c6a59822a8d96e38c44270dcd2d418, retrying]
May 08 13:39:59.848 - 670s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-08-115725: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e1eedec6b04495b7e6f218adec22341f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-45c6a59822a8d96e38c44270dcd2d418, retrying]

... 1 lines not shown

May 08 10:04:05.361 E ns/e2e-k8s-sig-apps-job-upgrade-8504 pod/foo-jwsjn node/ip-10-0-234-251.us-west-2.compute.internal uid/6667c8b2-a9fb-4618-9713-4ecb83d103b1 container/c reason/ContainerExit code/137 cause/Error
May 08 10:04:07.348 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-08-082220: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-35a24a007dc93946a9f15f5f6cbea991 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5a9bbf19aef7ca1ebb0e8932a9ade30e, retrying]
May 08 10:04:07.348 - 679s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-08-082220: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-35a24a007dc93946a9f15f5f6cbea991 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5a9bbf19aef7ca1ebb0e8932a9ade30e, retrying]

... 1 lines not shown

May 08 04:36:11.570 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-205-95.us-west-2.compute.internal uid/eba87052-43f6-4117-a163-8579543f3498 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-08T04:08:13.738006006Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.63.0, branch=rhaos-4.13-rhel-8, revision=7aaa0d9)"\nlevel=info ts=2024-05-08T04:08:13.738059224Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/arm64, user=root, date=20240507-18:10:03)"\nlevel=info ts=2024-05-08T04:08:13.738199039Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-08T04:08:18.169416767Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T04:08:18.169491663Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T04:09:47.611552721Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T04:17:34.596040336Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\n
May 08 04:36:20.320 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-arm64-2024-05-08-025233: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cdb2e1471b191381f4f13c5a3f45dceb expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6fad0973ccc6698591240b0256fb5ed1, retrying]
May 08 04:36:20.320 - 690s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-arm64-2024-05-08-025233: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cdb2e1471b191381f4f13c5a3f45dceb expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6fad0973ccc6698591240b0256fb5ed1, retrying]

... 1 lines not shown