Job:
periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-gcp-ovn-upgrade (all) - 13 runs, 23% failed, 433% of failures match = 100% impact
#1791185938517331968junit8 hours ago
May 16 21:07:28.382 E ns/openshift-e2e-loki pod/loki-promtail-md5zv node/ci-op-43mv6t9r-4ef92-mx9cf-worker-b-vhldm uid/1bfae1c6-c2d3-4f69-a584-77e34548443c container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 21:07:40.700 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-191307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3f7fc66a263dc7137e319e8da5620199 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-29448fa58b4387328117e74b79addebf, retrying]
May 16 21:07:40.700 - 572s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-191307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3f7fc66a263dc7137e319e8da5620199 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-29448fa58b4387328117e74b79addebf, retrying]

... 1 lines not shown

#1790698187401465856junit41 hours ago
May 15 12:52:06.887 - 7s    E clusteroperator/kube-storage-version-migrator condition/Available status/False reason/KubeStorageVersionMigratorAvailable: Waiting for Deployment
May 15 12:52:07.934 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-15-105412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8ebb58baf15bbf5591409eaa14167c12 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-524b3f7a9934ff5502c38e046bef72c9, retrying]
May 15 12:52:07.934 - 504s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-15-105412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8ebb58baf15bbf5591409eaa14167c12 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-524b3f7a9934ff5502c38e046bef72c9, retrying]

... 1 lines not shown

#1791096314721734656junit14 hours ago
May 16 15:08:19.111 E ns/openshift-multus pod/multus-additional-cni-plugins-8s4qf node/ci-op-19nvlp1i-4ef92-l4m5j-worker-b-wd948 uid/2ec7eeff-9ecb-4cd8-8660-546c8196e321 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 15:08:31.372 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-131307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-21de8e2a86688c1b70c88dae2a2eb8e4 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ecd6d2b0e7fec0bc532634e937d51cb0, retrying]
May 16 15:08:31.372 - 719s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-131307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-21de8e2a86688c1b70c88dae2a2eb8e4 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ecd6d2b0e7fec0bc532634e937d51cb0, retrying]

... 1 lines not shown

#1790977367259222016junit22 hours ago
May 16 07:17:28.185 E ns/openshift-multus pod/multus-additional-cni-plugins-sbtz4 node/ci-op-xctf51qs-4ef92-p4q62-worker-c-smtxg uid/6183ad53-64e7-409b-b026-be4de9b266c6 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 07:18:46.669 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-052445: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dd2c0b5981f20b1a6475d547122856e9 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-74bd12ab97da00b1cb01f30dbc4d93f2, retrying]
May 16 07:18:46.669 - 126s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-052445: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dd2c0b5981f20b1a6475d547122856e9 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-74bd12ab97da00b1cb01f30dbc4d93f2, retrying]

... 1 lines not shown

#1790292488137740288junit2 days ago
May 14 10:00:02.168 E ns/openshift-multus pod/multus-additional-cni-plugins-jdd7l node/ci-op-kmrd71cs-4ef92-8m7c4-worker-c-mq49t uid/a1fd45b2-60ce-4c00-a672-68d444b7e068 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 10:01:15.836 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-14-080341: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-00e5cf1a8d54634b18ae1b649ae07599 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-26726308d835bc141a80cef22a2b5ae0, retrying]
May 14 10:01:15.836 - 166s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-14-080341: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-00e5cf1a8d54634b18ae1b649ae07599 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-26726308d835bc141a80cef22a2b5ae0, retrying]

... 1 lines not shown

#1790788538875252736junit35 hours ago
May 15 18:44:40.532 E ns/openshift-cluster-node-tuning-operator pod/tuned-tgdtw node/ci-op-szc9cib5-4ef92-vchs4-worker-b-vrr8n uid/1acab8f3-2ec2-4d1f-a060-e5c80b41c654 container/tuned reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 15 18:44:40.545 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-15-165412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0e330ed47e604336d04ac618fbe6a014 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a8324daef6a6216785b859b4657fd17e, retrying]
May 15 18:44:40.545 - 611s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-15-165412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0e330ed47e604336d04ac618fbe6a014 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a8324daef6a6216785b859b4657fd17e, retrying]

... 1 lines not shown

#1788928530562682880junit6 days ago
May 10 15:36:41.137 E ns/openshift-network-diagnostics pod/network-check-target-6ccfs node/ci-op-0y0vbndi-4ef92-4r97f-worker-c-9qjwr uid/6425a912-c159-4684-99c5-abe384685e2d container/network-check-target-container reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 15:39:28.520 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-10-134150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f93b2c8241ba16b7f04c36b0ee62becc expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ae21261368f3a9af89304d1c58bd1865, retrying]
May 10 15:39:28.520 - 128s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-10-134150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f93b2c8241ba16b7f04c36b0ee62becc expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ae21261368f3a9af89304d1c58bd1865, retrying]

... 1 lines not shown

#1790384829783609344junit2 days ago
May 14 16:04:30.480 E ns/openshift-multus pod/multus-additional-cni-plugins-r49s8 node/ci-op-ims1wz5c-4ef92-s529q-worker-b-st849 uid/01c0df91-d6d1-4ddc-a0cf-86fd0713a6c6 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 16:04:54.831 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-14-140838: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fa63eec021d80b1a2b2e61c114c6debe expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-8454e9fa568057929a20510186666fcc, retrying]
May 14 16:04:54.831 - 654s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-14-140838: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fa63eec021d80b1a2b2e61c114c6debe expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-8454e9fa568057929a20510186666fcc, retrying]

... 1 lines not shown

#1788556419214086144junit7 days ago
May 09 14:52:49.502 E ns/openshift-monitoring pod/kube-state-metrics-584695477-5r2d8 node/ci-op-bgicl71m-4ef92-8lhj5-worker-c-ftrg7 uid/0cba50d9-9148-4f84-91cf-d64c86696560 container/kube-state-metrics reason/ContainerExit code/2 cause/Error
May 09 14:53:06.217 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-09-130302: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a794a60ad94d981e84d251529a289b78 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-adc313640347c65034662b415cfd8eb9, retrying]
May 09 14:53:06.217 - 513s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-09-130302: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a794a60ad94d981e84d251529a289b78 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-adc313640347c65034662b415cfd8eb9, retrying]

... 1 lines not shown

#1788233091991474176junit8 days ago
May 08 17:27:19.898 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ci-op-9j7r485g-4ef92-j5lbs-worker-c-tgxs2 uid/82325af2-2bf8-4308-8d7e-9ec8e13a343e container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-08T16:49:23.659950622Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.63.0, branch=rhaos-4.13-rhel-8, revision=7aaa0d9)"\nlevel=info ts=2024-05-08T16:49:23.660053168Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/amd64, user=root, date=20240507-18:09:48)"\nlevel=info ts=2024-05-08T16:49:23.660389013Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=error ts=2024-05-08T16:49:23.670877025Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9090/-/reload\": dial tcp [::1]:9090: connect: connection refused"\nlevel=info ts=2024-05-08T16:49:32.390573441Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T16:49:32.390727435Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T16:51:02.519661778Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T16:55:52.463913618Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/promethe
May 08 17:27:38.966 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-153833: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dc0d0136c3f603afce167331ad50d971 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d0a1ada846bd9a27ee3ce694d61591bc, retrying]
May 08 17:27:38.966 - 522s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-153833: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dc0d0136c3f603afce167331ad50d971 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d0a1ada846bd9a27ee3ce694d61591bc, retrying]

... 1 lines not shown

#1788135109744922624junit8 days ago
May 08 10:59:22.177 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-xi81qpj4-4ef92-cwsgp-worker-c-nxlp8 uid/6444b345-8cec-4839-aaff-30f109e75f67 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-08T10:19:03.573155149Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.63.0, branch=rhaos-4.13-rhel-8, revision=7aaa0d9)"\nlevel=info ts=2024-05-08T10:19:03.573254341Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/amd64, user=root, date=20240507-18:09:48)"\nlevel=info ts=2024-05-08T10:19:03.57356462Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-08T10:19:12.384373498Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T10:19:12.384539212Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T10:19:13.593812142Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T10:20:54.377916149Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T10:22:21.433389083Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/
May 08 10:59:27.854 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-090908: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-47f97956028c6c5f14e7b69b7a85df76 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-af561bc40e0013b66c1298bc2956e4ec, retrying]
May 08 10:59:27.854 - 675s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-090908: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-47f97956028c6c5f14e7b69b7a85df76 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-af561bc40e0013b66c1298bc2956e4ec, retrying]

... 1 lines not shown

#1788796252570259456junit6 days ago
May 10 06:58:17.298 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-6l6mj node/ci-op-c3w4xqd3-4ef92-bntdd-worker-c-j8czk uid/35eef3e9-cc45-4149-bb6a-0242cf64e463 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 10 07:00:19.080 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-10-045757: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8f89b7e710618ebb5786a63ce873de39 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9813482eef09dbb9da62e1b866e48543, retrying]
May 10 07:00:19.080 - 161s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-10-045757: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8f89b7e710618ebb5786a63ce873de39 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9813482eef09dbb9da62e1b866e48543, retrying]

... 1 lines not shown

#1788039914789539840junit9 days ago
May 08 04:48:26.372 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-pqbfc node/ci-op-m9iitbpk-4ef92-lgq9w-worker-b-twq46 uid/50f0054f-2edc-4661-932c-86557b59520e container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 08 04:49:45.099 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-025210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7df0814bfa268b0f80beeacfce8476e2 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a56499fb93a595baf559caace82a288c, retrying]
May 08 04:49:45.099 - 232s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-025210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7df0814bfa268b0f80beeacfce8476e2 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a56499fb93a595baf559caace82a288c, retrying]

... 1 lines not shown

Found in 100.00% of runs (433.33% of failures) across 13 total runs and 1 jobs (23.08% failed) in 138ms - clear search | chart view - source code located on github