May 16 21:02:51.494 E ns/openshift-monitoring pod/alertmanager-main-1 node/ci-op-p1qrd1xg-da17d-cnf26-worker-b-pw2pf uid/d9a1a2d1-d528-45e2-9abf-2f5a53ce4d62 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-16T21:02:49.133Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=cca15d1d9be3ebe61f12f5179e54f0c4c8b23e8b)"\nts=2024-05-16T21:02:49.133Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@0efa572a5d00, date=20240515-05:16:57)"\nts=2024-05-16T21:02:49.198Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-16T21:02:49.293Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-16T21:02:49.293Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-16T21:02:49.293Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=95.57534ms\n
May 16 21:02:58.490 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-191307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f30d4779b7c772f683f9208b04cebdeb expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-aeea3f0051280effc492fedb2e0fbc52, retrying]
May 16 21:02:58.490 - 596s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-191307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f30d4779b7c772f683f9208b04cebdeb expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-aeea3f0051280effc492fedb2e0fbc52, retrying]

... 1 lines not shown

May 15 18:27:59.682 E ns/openshift-multus pod/network-metrics-daemon-6lr7t node/ci-op-vsvcfg79-da17d-89lwr-worker-b-g8ldk uid/6d42b3be-4653-4ed7-813b-bb2b08b3462e container/network-metrics-daemon reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 18:28:02.018 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-15-165412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c3965ede80debb18cc4319859aa20faa expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e6153e5e36c66ff40a9f07372fb70b73, retrying]
May 15 18:28:02.018 - 600s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-15-165412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c3965ede80debb18cc4319859aa20faa expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e6153e5e36c66ff40a9f07372fb70b73, retrying]

... 1 lines not shown

May 15 12:40:45.529 E ns/openshift-multus pod/multus-additional-cni-plugins-stkjn node/ci-op-sdrcmwxp-da17d-2cxl8-worker-c-xgfnn uid/c10fbc0e-4363-4fe6-bd9e-fe2f5d3d38da container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 12:42:41.616 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-15-105412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3b624b2bdd366d89f3586bd3dc2833fa expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bda5ba44829dd36684021621c732a5e7, retrying]
May 15 12:42:41.616 - 145s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-15-105412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3b624b2bdd366d89f3586bd3dc2833fa expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bda5ba44829dd36684021621c732a5e7, retrying]

... 1 lines not shown

May 16 07:03:14.152 E ns/openshift-multus pod/multus-additional-cni-plugins-prwg2 node/ci-op-p258kkls-da17d-frhpt-worker-c-xsvkj uid/2e00d828-4939-4ece-aa6c-c2c4a25be2d3 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 07:06:01.899 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-052445: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-644eec8603e0231f561d5bc3b42667f6 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-984475778e2d04d6418b15ea143104c2, retrying]
May 16 07:06:01.899 - 182s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-052445: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-644eec8603e0231f561d5bc3b42667f6 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-984475778e2d04d6418b15ea143104c2, retrying]

... 1 lines not shown

May 10 06:33:59.977 - 999ms E ns/openshift-console route/console disruption/ingress-to-console connection/reused reason/DisruptionBegan ns/openshift-console route/console disruption/ingress-to-console connection/reused stopped responding to GET requests over reused connections: Get "https://console-openshift-console.apps.ci-op-fiwhn6n8-da17d.XXXXXXXXXXXXXXXXXXXXXX/healthz": net/http: timeout awaiting response headers
May 10 06:35:48.720 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-10-045757: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3deaa8889030321870de8762fe688faf expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-254f13f59bd0748d06bde4ea9a4cd425, retrying]
May 10 06:35:48.720 - 139s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-10-045757: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3deaa8889030321870de8762fe688faf expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-254f13f59bd0748d06bde4ea9a4cd425, retrying]

... 1 lines not shown

May 14 15:50:18.398 - 9s    E clusteroperator/dns condition/Degraded status/True reason/DNS default is degraded
May 14 15:52:52.963 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-14-140838: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cf5dfb68e4af7e20de978bfed97787fd expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-94ba25e984adf9b4e950bf7c44be2ab7, retrying]
May 14 15:52:52.963 - 160s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-14-140838: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cf5dfb68e4af7e20de978bfed97787fd expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-94ba25e984adf9b4e950bf7c44be2ab7, retrying]

... 1 lines not shown

May 10 15:25:20.854 E ns/openshift-multus pod/multus-additional-cni-plugins-gfnj5 node/ci-op-f2xh9s65-da17d-778tn-worker-c-g8ns6 uid/981d0981-6cbb-4524-9d6d-4304eb5e1b14 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 15:27:54.811 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-10-134150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-24f5a5e3ed1fbfd356285db6a2a60a0f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2d7428f9183bc51bdad08ef9994873b6, retrying]
May 10 15:27:54.811 - 173s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-10-134150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-24f5a5e3ed1fbfd356285db6a2a60a0f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2d7428f9183bc51bdad08ef9994873b6, retrying]

... 1 lines not shown

May 08 17:21:48.581 - 999ms E ns/openshift-console route/console disruption/ingress-to-console connection/reused reason/DisruptionBegan ns/openshift-console route/console disruption/ingress-to-console connection/reused stopped responding to GET requests over reused connections: Get "https://console-openshift-console.apps.ci-op-7y9v06lk-da17d.XXXXXXXXXXXXXXXXXXXXXX/healthz": net/http: timeout awaiting response headers
May 08 17:23:08.194 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-153833: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c5c1a2188c6b886354a9bcc9923268de expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-00e88925a8997cc6d538a1c1e4099794, retrying]
May 08 17:23:08.194 - 81s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-153833: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c5c1a2188c6b886354a9bcc9923268de expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-00e88925a8997cc6d538a1c1e4099794, retrying]

... 1 lines not shown

May 08 10:49:25.331 E ns/e2e-k8s-sig-apps-job-upgrade-6320 pod/foo-d7vbz node/ci-op-qdyiz73x-da17d-hr7ld-worker-c-fwp48 uid/7d6ae3a4-5fec-4ec8-a9bf-18bf92c76926 container/c reason/ContainerExit code/137 cause/Error
May 08 10:49:53.517 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-090908: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d627d09cd45512ab65846f2ced131b44 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ac7189a48b39f3ecde8dcc7b8ef8dd34, retrying]
May 08 10:49:53.517 - 569s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-090908: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d627d09cd45512ab65846f2ced131b44 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ac7189a48b39f3ecde8dcc7b8ef8dd34, retrying]

... 1 lines not shown

May 08 04:29:22.481 E ns/openshift-multus pod/multus-admission-controller-5cd68b7cf5-vx2pv node/ci-op-6qn396g8-da17d-z456t-master-2 uid/a4e2f4fc-4b73-447f-8ec5-ffddfe4f9093 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 08 04:31:28.528 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-025210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e894cfdf9dfc1270ebe1656aa7a6b303 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6ed1bfecce9ac9127357ad296c5261f3, retrying]
May 08 04:31:28.528 - 336s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-025210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e894cfdf9dfc1270ebe1656aa7a6b303 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6ed1bfecce9ac9127357ad296c5261f3, retrying]

... 1 lines not shown