May 16 21:48:14.509 E ns/openshift-monitoring pod/alertmanager-main-0 node/libvirt-s390x-2-0-f88-qhs96-worker-0-wp2zg uid/7c495b15-fe41-474b-858d-d45327a6e7ae container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-16T21:48:11.570Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=cca15d1d9be3ebe61f12f5179e54f0c4c8b23e8b)"\nts=2024-05-16T21:48:11.570Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@d6a3c5a3ff7c, date=20240515-05:16:26)"\nts=2024-05-16T21:48:11.607Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-16T21:48:11.664Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-16T21:48:11.664Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-16T21:48:11.665Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=57.962586ms\n
May 16 21:49:07.217 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-16-190041: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c52239f35ae6b64d2d2b40c494b5343a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ac274c44a212eb373f764f5c98849b34, retrying]
May 16 21:49:07.217 - 519s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-16-190041: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c52239f35ae6b64d2d2b40c494b5343a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ac274c44a212eb373f764f5c98849b34, retrying]

... 1 lines not shown

May 15 09:50:20.700 E ns/openshift-marketplace pod/certified-operators-5cr57 node/libvirt-s390x-3-0-f88-twhkd-master-0 uid/1dba95b8-19a3-4ef8-bce4-9642c40866ab container/registry-server reason/ContainerExit code/2 cause/Error time="2024-05-15T09:50:17Z" level=info msg="starting pprof endpoint" address="localhost:6060"\n
May 15 09:52:23.693 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-14-130501: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-776e916f44a4e7e84b1da09f3f7db94e expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-32d8c2ff0e45a5f4cada06789d85cefb, retrying]
May 15 09:52:23.693 - 210s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-14-130501: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-776e916f44a4e7e84b1da09f3f7db94e expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-32d8c2ff0e45a5f4cada06789d85cefb, retrying]

... 1 lines not shown

May 16 09:49:04.969 E ns/openshift-multus pod/multus-admission-controller-c5d5ffb9-6nszk node/libvirt-s390x-5-1-f88-956h9-master-0 uid/8c572a9c-9bf0-4ee8-b9d3-7e2256f04bca container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 16 09:51:41.460 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-16-052448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a1c053bf14a616b914418158bd021576 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fdc68250c6ab454be2477957af502730, retrying]
May 16 09:51:41.460 - 196s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-16-052448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a1c053bf14a616b914418158bd021576 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fdc68250c6ab454be2477957af502730, retrying]

... 1 lines not shown

May 15 21:47:50.733 E ns/openshift-multus pod/multus-additional-cni-plugins-gl9lg node/libvirt-s390x-3-1-f88-5mbvd-worker-0-7qxtr uid/91d6ce2f-5a69-474e-9197-9eca04aee99c container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 21:48:03.224 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-15-124235: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-214b2326824c49036b726f58b21015d5 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b52375d5b962a5ef99fac1e5bd0a1d27, retrying]
May 15 21:48:03.224 - 780s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-15-124235: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-214b2326824c49036b726f58b21015d5 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b52375d5b962a5ef99fac1e5bd0a1d27, retrying]

... 1 lines not shown

May 14 09:58:00.395 - 3s    E ns/openshift-authentication route/oauth-openshift disruption/ingress-to-oauth-server connection/new reason/DisruptionBegan ns/openshift-authentication route/oauth-openshift disruption/ingress-to-oauth-server connection/new stopped responding to GET requests over new connections: Get "https://oauth-openshift.apps.libvirt-s390x-4-0-f885a.libvirt-s390x-4-0.ci/healthz": net/http: TLS handshake timeout
May 14 09:58:25.672 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-10-124237: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7290127061bf651663ee6c7b89cefd08 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-d6ee5c45b135739f59dc341a55bb68d6, retrying]
May 14 09:58:25.672 - 183s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-10-124237: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7290127061bf651663ee6c7b89cefd08 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-d6ee5c45b135739f59dc341a55bb68d6, retrying]

... 1 lines not shown

May 14 21:38:06.651 E ns/openshift-multus pod/multus-additional-cni-plugins-qcj2r node/libvirt-s390x-2-1-f88-jqlbf-worker-0-7nqdc uid/eb52dfb0-2078-4605-b22e-f4273bcd3c5f container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 21:39:22.318 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-14-130501: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1b7f916be4cbff92fb5703833bebc5cf expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-44c0b7f57b87b441536f70eaeda7f9fc, retrying]
May 14 21:39:22.318 - 462s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-14-130501: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1b7f916be4cbff92fb5703833bebc5cf expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-44c0b7f57b87b441536f70eaeda7f9fc, retrying]

... 1 lines not shown

May 13 22:12:08.000 - 4s    E ns/openshift-image-registry route/test-disruption-new disruption/image-registry connection/new reason/DisruptionBegan ns/openshift-image-registry route/test-disruption-new disruption/image-registry connection/new stopped responding to GET requests over new connections: Get "https://test-disruption-new-openshift-image-registry.apps.libvirt-s390x-4-0-f885a.libvirt-s390x-4-0.ci/healthz": net/http: TLS handshake timeout
May 13 22:12:19.949 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-10-124237: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-322d16d6fe4eb1b170f4b022bbab5834 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-19de700d308f6df0aadb765ef02ff1c6, retrying]
May 13 22:12:19.949 - 361s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-10-124237: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-322d16d6fe4eb1b170f4b022bbab5834 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-19de700d308f6df0aadb765ef02ff1c6, retrying]

... 1 lines not shown

May 10 21:57:12.086 - 1s    E node/libvirt-s390x-4-0-f88-ltdbb-master-0 reason/FailedToDeleteCGroupsPath May 10 21:57:12.086466 libvirt-s390x-4-0-f88-ltdbb-master-0 kubenswrapper[2251]: I0510 21:57:12.086395    2251 pod_container_manager_linux.go:191] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pod795b14a8-809d-4ad1-b90c-73fe44bfcdfc] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod795b14a8-809d-4ad1-b90c-73fe44bfcdfc] : Timed out while waiting for systemd to remove kubepods-burstable-pod795b14a8_809d_4ad1_b90c_73fe44bfcdfc.slice"
May 10 22:00:38.894 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-10-124237: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7b5352919178b84b0411725a025f20ed expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e3205e817e26b10347f8627347b14b32, retrying]
May 10 22:00:38.894 - 95s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-10-124237: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7b5352919178b84b0411725a025f20ed expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e3205e817e26b10347f8627347b14b32, retrying]

... 1 lines not shown

May 10 09:51:55.020 E ns/openshift-multus pod/multus-admission-controller-65bd58db4d-rw4sd node/libvirt-s390x-2-1-f88-ldgxc-master-1 uid/9e83795c-b293-41b6-a8f9-160be29f647c container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 09:53:16.344 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-10-045742: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5249186312bcb6596f98003c3d4e5024 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-63dfb74eafee5bb12b1c3b1db0cdd14d, retrying]
May 10 09:53:16.344 - 836s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-10-045742: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5249186312bcb6596f98003c3d4e5024 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-63dfb74eafee5bb12b1c3b1db0cdd14d, retrying]

... 1 lines not shown

May 09 21:54:45.568 E ns/openshift-monitoring pod/alertmanager-main-1 node/libvirt-s390x-3-1-f88-wsnwm-worker-0-qshjl uid/b33fbf18-5bea-4c79-bef3-c6a63e44a82e container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-09T21:54:41.625Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=c5f095c646ea4c1f9524833e8e3579b827a501aa)"\nts=2024-05-09T21:54:41.626Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@a9f5bd1cf77c, date=20240425-19:11:23)"\nts=2024-05-09T21:54:41.793Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-09T21:54:41.950Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-09T21:54:41.950Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-09T21:54:41.950Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=156.92605ms\n
May 09 21:55:17.275 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-09-124016: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2b3273476881a2082c3c9bf746512807 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7174c50c01671697ce2a75f3674b1f22, retrying]
May 09 21:55:17.275 - 606s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-09-124016: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2b3273476881a2082c3c9bf746512807 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7174c50c01671697ce2a75f3674b1f22, retrying]

... 1 lines not shown

May 09 09:49:02.135 - 772s  E alert/Watchdog ns/openshift-monitoring ALERTS{alertname="Watchdog", alertstate="firing", namespace="openshift-monitoring", prometheus="openshift-monitoring/k8s", severity="none"}
May 09 09:50:03.488 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-08-174141: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fda461a0069cf9a0503111d4c3304696 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3b38f71a531b8c984da758fb7f38d0f0, retrying]
May 09 09:50:03.488 - 528s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-08-174141: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fda461a0069cf9a0503111d4c3304696 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3b38f71a531b8c984da758fb7f38d0f0, retrying]

... 1 lines not shown

May 08 21:57:47.216 E ns/openshift-monitoring pod/alertmanager-main-0 node/libvirt-s390x-5-0-f88-wx4mh-worker-0-lphd7 uid/cd35667e-1d1c-4f10-a581-df141df81274 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-08T21:57:41.871Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=c5f095c646ea4c1f9524833e8e3579b827a501aa)"\nts=2024-05-08T21:57:41.871Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@a9f5bd1cf77c, date=20240425-19:11:23)"\nts=2024-05-08T21:57:41.964Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-08T21:57:42.190Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-08T21:57:42.200Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-08T21:57:42.200Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=235.697919ms\n
May 08 21:58:11.568 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-08-174141: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0b147d2f7a5d9a7ec887b458f5f7fe0e expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-73671a1fd3ba62dff3308449423a5876, retrying]
May 08 21:58:11.568 - 638s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-08-174141: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0b147d2f7a5d9a7ec887b458f5f7fe0e expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-73671a1fd3ba62dff3308449423a5876, retrying]

... 1 lines not shown

May 08 09:52:14.628 - 2906s E alert/Watchdog ns/openshift-monitoring ALERTS{alertname="Watchdog", alertstate="firing", namespace="openshift-monitoring", prometheus="openshift-monitoring/k8s", severity="none"}
May 08 09:53:20.696 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-08-025150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-600d0d046a65e8f0aeb2c370d481f821 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-7e60fa8d1ffddfcdd838207dacb165f0, retrying]
May 08 09:53:20.696 - 2597s E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-08-025150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-600d0d046a65e8f0aeb2c370d481f821 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-7e60fa8d1ffddfcdd838207dacb165f0, retrying]

... 1 lines not shown

May 07 21:51:54.474 E ns/openshift-monitoring pod/alertmanager-main-0 node/libvirt-s390x-4-1-f88-7dnzk-worker-0-gwcmx uid/b3e1cc22-ad10-4c0d-aa6c-7cae0a615114 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-07T21:51:51.661Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=rhaos-4.13-rhel-8, revision=c5f095c646ea4c1f9524833e8e3579b827a501aa)"\nts=2024-05-07T21:51:51.661Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@a9f5bd1cf77c, date=20240425-19:11:23)"\nts=2024-05-07T21:51:51.692Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-07T21:51:51.745Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-07T21:51:51.745Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-07T21:51:51.745Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=52.462466ms\n
May 07 21:52:48.764 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9d484c1aba21fd730a43ee45e5fe4197 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-663b95583fbccf4d272fc615ad1390b7, retrying]
May 07 21:52:48.764 - 631s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9d484c1aba21fd730a43ee45e5fe4197 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-663b95583fbccf4d272fc615ad1390b7, retrying]

... 1 lines not shown

May 07 09:57:49.637 E ns/openshift-multus pod/multus-admission-controller-69f66d4565-6jb9g node/libvirt-s390x-3-1-f88-d6g56-master-1 uid/19141ea4-317b-4d03-abfa-265d8b1b0f8d container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 07 10:00:19.768 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bb73420309bbd5a21a0add8fc2c6bbae expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-dc405b95ed0717949a82b634b95bb8af, retrying]
May 07 10:00:19.768 - 182s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bb73420309bbd5a21a0add8fc2c6bbae expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-dc405b95ed0717949a82b634b95bb8af, retrying]

... 1 lines not shown

May 06 21:50:36.954 E ns/openshift-multus pod/multus-admission-controller-69f66d4565-hkzzk node/libvirt-s390x-3-0-f88-nwlv5-master-2 uid/1dbdb0ee-21ab-4e07-90b9-cfd91052c15e container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 06 21:52:42.659 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-280c91ddc3af1cd17391eafed2034d09 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e24f241a21c7b6707619ddebf8b2ae8b, retrying]
May 06 21:52:42.659 - 748s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-280c91ddc3af1cd17391eafed2034d09 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e24f241a21c7b6707619ddebf8b2ae8b, retrying]

... 1 lines not shown

May 06 10:17:36.299 E ns/openshift-marketplace pod/redhat-operators-grhkd node/libvirt-s390x-6-1-f88-6tr8t-master-0 uid/ee345528-b71c-4493-8dc7-eca2d2e344e8 container/registry-server reason/ContainerExit code/137 cause/Error time="2024-05-06T10:05:11Z" level=info msg="starting pprof endpoint" address="localhost:6060"\ntime="2024-05-06T10:05:19Z" level=info msg="serving registry" configs=/configs port=50051\ntime="2024-05-06T10:05:19Z" level=info msg="stopped caching cpu profile data" address="localhost:6060"\ntime="2024-05-06T10:16:58Z" level=info msg="shutting down..." configs=/configs port=50051\n
May 06 10:20:59.132 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6b6e3103901b04641824489be1b38a86 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ddef661219fc14b1f36ea214011c94e6, retrying]
May 06 10:20:59.132 - 164s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6b6e3103901b04641824489be1b38a86 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ddef661219fc14b1f36ea214011c94e6, retrying]

... 1 lines not shown

May 03 21:47:54.276 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-mzd28 node/libvirt-s390x-5-0-f88-mvddm-worker-0-sdnh6 uid/d1890670-5f3b-45da-8cfd-f9c0d1ccdd2f container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 03 21:49:47.249 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-464b2cbc69e84031da1904077ba90790 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a5f2a7373c9eac300fa4a0eb0e1fa733, retrying]
May 03 21:49:47.249 - 180s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-s390x-2024-05-02-145021: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-464b2cbc69e84031da1904077ba90790 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a5f2a7373c9eac300fa4a0eb0e1fa733, retrying]

... 1 lines not shown