May 16 20:59:08.081 - 999ms E disruption/oauth-api connection/reused reason/DisruptionBegan disruption/oauth-api connection/reused stopped responding to GET requests over reused connections: error running request: 500 Internal Server Error: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Unknown desc = malformed header: missing HTTP content-type","code":500}\n
May 16 20:59:26.123 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-16-190038: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9b1611abf2a8f4414a2febef65d98f6a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-54c99b5c2ff4cc12b5699d3313044908, retrying]
May 16 20:59:26.123 - 562s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-16-190038: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9b1611abf2a8f4414a2febef65d98f6a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-54c99b5c2ff4cc12b5699d3313044908, retrying]

... 1 lines not shown

May 15 14:39:24.222 E ns/openshift-multus pod/multus-additional-cni-plugins-dzbr2 node/ip-10-0-129-199.us-west-1.compute.internal uid/f4c0510e-fe07-4035-8931-9fd27f401a24 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 14:39:41.020 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-15-124232: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-027bc332c16c41303ad2ac7e63fb715e expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a5e3572fff83925339f0ceae3ab083fd, retrying]
May 15 14:39:41.020 - 690s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-15-124232: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-027bc332c16c41303ad2ac7e63fb715e expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a5e3572fff83925339f0ceae3ab083fd, retrying]

... 1 lines not shown

May 16 15:26:55.427 E ns/openshift-e2e-loki pod/event-exporter-6bdb7d5ddf-5bccj node/ip-10-0-152-126.us-west-1.compute.internal uid/802056ab-05fa-4348-b714-5d4194676aea container/event-exporter reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 16 15:26:56.780 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-16-131307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d0d611464fe0c4c5b301f6ca36fa4925 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6483946215a9ca57724a8829bc222f1f, retrying]
May 16 15:26:56.780 - 278s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-16-131307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d0d611464fe0c4c5b301f6ca36fa4925 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6483946215a9ca57724a8829bc222f1f, retrying]

... 1 lines not shown

May 16 07:28:16.875 E ns/openshift-multus pod/multus-additional-cni-plugins-c8sr4 node/ip-10-0-207-58.us-west-2.compute.internal uid/03013506-b60c-4f73-9ce9-30597db9b4ea container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 07:28:39.257 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-16-052445: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5ec441e6e8f6d5581847900ccf83eee1 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-854e0306309f95e321236b6ed86ad0b2, retrying]
May 16 07:28:39.257 - 704s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-16-052445: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5ec441e6e8f6d5581847900ccf83eee1 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-854e0306309f95e321236b6ed86ad0b2, retrying]

... 1 lines not shown

May 15 13:10:31.080 - 99s   E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 15 13:10:35.197 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-15-105242: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e6e8920af11351c1c6c1036c3fa72f99 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-47c480d44d0717942f49fbd3183575e2, retrying]
May 15 13:10:35.197 - 135s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-15-105242: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e6e8920af11351c1c6c1036c3fa72f99 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-47c480d44d0717942f49fbd3183575e2, retrying]

... 1 lines not shown

May 14 11:51:07.971 E ns/e2e-k8s-sig-apps-job-upgrade-490 pod/foo-9prt8 node/ip-10-0-131-177.ec2.internal uid/0a522546-3660-481d-a520-7863b683228f container/c reason/ContainerExit code/137 cause/Error
May 14 11:52:56.361 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-14-094405: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b2d1ecba140ade12a209a6f6f602e067 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-9c5eb07a040b11f904e1af238c08c319, retrying]
May 14 11:52:56.361 - 545s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-14-094405: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b2d1ecba140ade12a209a6f6f602e067 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-9c5eb07a040b11f904e1af238c08c319, retrying]

... 1 lines not shown

May 14 15:16:44.657 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-154-141.us-west-2.compute.internal uid/83a5babf-58f1-46b7-9493-fcdce6a0c7e7 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-14T15:16:43.562Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=38194c1bde7b693bb84526d1084400a1e76da31b)"\nts=2024-05-14T15:16:43.562Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@cbb9d02780b4, date=20240510-00:07:28)"\nts=2024-05-14T15:16:43.593Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-14T15:16:43.622Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-14T15:16:43.622Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-14T15:16:43.622Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=29.169365ms\n
May 14 15:16:45.317 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-14-130501: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2a79c4390db331f942b870566850d684 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b859269e3bb86f8c0db92f67cebdee5c, retrying]
May 14 15:16:45.317 - 710s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-14-130501: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2a79c4390db331f942b870566850d684 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b859269e3bb86f8c0db92f67cebdee5c, retrying]

... 1 lines not shown

May 11 05:20:20.814 E ns/e2e-k8s-sig-apps-job-upgrade-55 pod/foo-hxnsp node/ip-10-0-251-131.us-west-2.compute.internal uid/c04e0c0c-0cfa-478a-886c-a95530b24693 container/c reason/ContainerExit code/137 cause/Error
May 11 05:20:40.522 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-10-124320: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3b9e56c3f15bcfc2e9d69e08b03bea4f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a846285122e8e2c3a01e7a206ddb72df, retrying]
May 11 05:20:40.522 - 653s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-10-124320: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3b9e56c3f15bcfc2e9d69e08b03bea4f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a846285122e8e2c3a01e7a206ddb72df, retrying]

... 1 lines not shown

May 14 10:08:51.552 - 103s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 14 10:09:03.368 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-14-080341: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b9a59725782608ae61cbeffa8db564cd expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6f262ab3ba51c2bc9bf4e0581b23508a, retrying]
May 14 10:09:03.368 - 131s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-14-080341: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b9a59725782608ae61cbeffa8db564cd expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6f262ab3ba51c2bc9bf4e0581b23508a, retrying]

... 1 lines not shown

May 10 14:44:08.226 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-160-16.ec2.internal uid/bbea4217-274e-49b0-a426-de8c884e64ab container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-10T14:44:05.905Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=38194c1bde7b693bb84526d1084400a1e76da31b)"\nts=2024-05-10T14:44:05.905Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@cb558022914e, date=20240510-00:07:09)"\nts=2024-05-10T14:44:05.934Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-10T14:44:05.970Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-10T14:44:05.970Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-10T14:44:05.970Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=36.384598ms\n
May 10 14:45:27.877 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-10-124320: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d1e987a1ff0818800327607f8b9c3aa5 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8bca4fc1b0fad82c62717fcb639b6019, retrying]
May 10 14:45:27.877 - 204s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-10-124320: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d1e987a1ff0818800327607f8b9c3aa5 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8bca4fc1b0fad82c62717fcb639b6019, retrying]

... 1 lines not shown

May 10 11:36:06.286 E ns/openshift-multus pod/multus-admission-controller-6c977b77db-4xvfd node/ip-10-0-136-132.us-west-1.compute.internal uid/42d8f836-6e20-43ac-a8d9-8d44c9455c2c container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 11:38:39.806 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-10-092528: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9e7f22a83a514eb62f86b7145c7298f8 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9460d9fb928a636c4f4ba4ca58979420, retrying]
May 10 11:38:39.806 - 226s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-10-092528: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9e7f22a83a514eb62f86b7145c7298f8 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9460d9fb928a636c4f4ba4ca58979420, retrying]

... 1 lines not shown

May 10 12:40:03.668 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-167-14.us-west-1.compute.internal uid/2b0e3486-d611-42ea-a5d0-7e1531a9e61e container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-10T12:39:54.054Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=38194c1bde7b693bb84526d1084400a1e76da31b)"\nts=2024-05-10T12:39:54.055Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@cb558022914e, date=20240510-00:07:09)"\nts=2024-05-10T12:39:54.075Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-10T12:39:54.108Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-10T12:39:54.108Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-10T12:39:54.108Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=33.188267ms\n
May 10 12:41:43.404 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-10-102839: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f5ca70fae4d54e71a1093d2482655625 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f8a868105de61e79f92bce6afe9364d3, retrying]
May 10 12:41:43.404 - 246s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-10-102839: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f5ca70fae4d54e71a1093d2482655625 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f8a868105de61e79f92bce6afe9364d3, retrying]

... 1 lines not shown

May 10 07:17:34.113 E ns/openshift-ingress-canary pod/ingress-canary-4txdn node/ip-10-0-154-193.us-west-2.compute.internal uid/bea608a0-5a33-4265-a393-ea3d09f52521 container/serve-healthcheck-canary reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 07:17:34.437 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-10-045721: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f85d030face5f056dbd167713857b480 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e08441f88d241595d6a8a34395afa287, retrying]
May 10 07:17:34.437 - 258s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-10-045721: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f85d030face5f056dbd167713857b480 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e08441f88d241595d6a8a34395afa287, retrying]

... 1 lines not shown

May 09 14:43:24.194 E ns/e2e-k8s-sig-apps-job-upgrade-2470 pod/foo-l7869 node/ip-10-0-194-53.us-west-2.compute.internal uid/2683d92a-6fde-4751-bd44-48da30c7e824 container/c reason/ContainerExit code/137 cause/Error
May 09 14:43:39.374 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-09-124058: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-af75ff0b3d4a670aa4c70bc19e920da4 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-4fbd6133001fb32d377352cf5cf449d5, retrying]
May 09 14:43:39.374 - 665s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-09-124058: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-af75ff0b3d4a670aa4c70bc19e920da4 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-4fbd6133001fb32d377352cf5cf449d5, retrying]

... 1 lines not shown

May 08 19:52:23.664 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-136-4.us-west-1.compute.internal uid/b5c5a9dd-434e-4da2-b7d0-1423cc22d422 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T19:52:21.075Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=c5f095c646ea4c1f9524833e8e3579b827a501aa)"\nts=2024-05-08T19:52:21.075Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@3116edadf533, date=20240425-19:09:55)"\nts=2024-05-08T19:52:21.101Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T19:52:21.137Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T19:52:21.137Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T19:52:21.137Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=35.816962ms\n
May 08 19:53:24.028 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-08-174222: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-331c6671c63a1530d227cbdea5a81c36 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bae61650da040a535f468d7a415ba019, retrying]
May 08 19:53:24.028 - 244s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-08-174222: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-331c6671c63a1530d227cbdea5a81c36 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bae61650da040a535f468d7a415ba019, retrying]

... 1 lines not shown

May 08 13:54:38.136 - 123s  E clusteroperator/control-plane-machine-set condition/Available status/False reason/Missing 1 available replica(s)
May 08 13:54:50.476 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-08-115804: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-25cf027e8e729ebb82453967fe364599 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6b09bccb20a4243be31b83141fc23d0e, retrying]
May 08 13:54:50.476 - 542s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-08-115804: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-25cf027e8e729ebb82453967fe364599 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6b09bccb20a4243be31b83141fc23d0e, retrying]

... 1 lines not shown

May 08 10:41:36.608 E ns/openshift-monitoring pod/kube-state-metrics-5f6d49648-d8lnk node/ip-10-0-128-126.us-west-2.compute.internal uid/2b555f38-d0c1-48d7-aced-2f9a89497d26 container/kube-state-metrics reason/ContainerExit code/2 cause/Error
May 08 10:41:40.379 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-08-082259: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8a702db38c9050ebdfcd4adafc57ee8a expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ee54322e812ef5edaf3f38d2d20bb65c, retrying]
May 08 10:41:40.379 - 706s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-08-082259: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8a702db38c9050ebdfcd4adafc57ee8a expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ee54322e812ef5edaf3f38d2d20bb65c, retrying]

... 1 lines not shown

May 08 04:57:41.000 - 1s    E disruption/service-load-balancer-with-pdb connection/new reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/new stopped responding to GET requests over new connections: Get "http://a12cd779dd0464afe846abf11e6bfb51-1875439735.us-east-2.elb.amazonaws.com:80/echo?msg=Hello": read tcp 10.130.171.168:52922->13.59.224.127:80: read: connection reset by peer
May 08 04:57:42.865 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-08-025127: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dd331592f6b54a5192952ea2e7810479 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8d23d8f6650c43b4ea6120dc0ab23b44, retrying]
May 08 04:57:42.865 - 127s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-08-025127: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dd331592f6b54a5192952ea2e7810479 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8d23d8f6650c43b4ea6120dc0ab23b44, retrying]

... 1 lines not shown

May 04 05:37:10.755 E ns/openshift-monitoring pod/alertmanager-main-1 node/ip-10-0-215-231.us-west-2.compute.internal uid/8a8afe4a-b5ad-46ae-909d-2aefbe80341a container/alertmanager reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 04 05:37:20.795 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-multi-2024-05-02-145106: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-26f9de69c99418092fc0968bb791072b expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-89f05e229dac5e1bdf095e3bff210cc6, retrying]
May 04 05:37:20.795 - 238s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-multi-2024-05-02-145106: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-26f9de69c99418092fc0968bb791072b expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-89f05e229dac5e1bdf095e3bff210cc6, retrying]

... 1 lines not shown