May 16 17:32:47.473 E ns/openshift-monitoring pod/node-exporter-8qcm6 node/worker-1.ostest.test.metalkube.org uid/e41331d7-baa1-43a7-b9bb-b1cec0164ad0 container/kube-rbac-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 17:32:50.278 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-16-144840: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a386338f9618e73fdfdcd0676f3c3e89 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a26817e4b7f83563d8ac42e4175b7016, retrying]
May 16 17:32:50.278 - 236s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-16-144840: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a386338f9618e73fdfdcd0676f3c3e89 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a26817e4b7f83563d8ac42e4175b7016, retrying]

... 1 lines not shown

May 15 23:27:50.372 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-d9zl8 node/worker-0.ostest.test.metalkube.org uid/8a59a446-d92a-413b-b360-623890da26a7 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 15 23:28:46.774 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-15-204102: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1be471bd12042d2886b188efb0902fa0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-21a1b6fcecb3e8d9e6359a90b347bdb7, retrying]
May 15 23:28:46.774 - 614s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-15-204102: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1be471bd12042d2886b188efb0902fa0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-21a1b6fcecb3e8d9e6359a90b347bdb7, retrying]

... 1 lines not shown

May 11 17:08:44.020 E ns/openshift-kni-infra pod/haproxy-master-0.ostest.test.metalkube.org node/master-0.ostest.test.metalkube.org uid/77077710-45ad-4899-ad28-e20a29329917 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 11 17:09:17.676 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-11-142650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9d802504783062d2ebe167ca884ac014 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f710a84c3e62506111006447337d6a03, retrying]
May 11 17:09:17.676 - 682s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-11-142650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9d802504783062d2ebe167ca884ac014 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f710a84c3e62506111006447337d6a03, retrying]

... 1 lines not shown

May 11 11:14:28.749 - 999ms E disruption/cache-openshift-api connection/reused reason/DisruptionBegan disruption/cache-openshift-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ostest.test.metalkube.org:6443/apis/image.openshift.io/v1/namespaces/default/imagestreams?resourceVersion=0": net/http: timeout awaiting response headers
May 11 11:15:53.069 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-11-082650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d50a615bc4172489a74c679b5e0894b7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f71fc915593dcd05a47c66c45de7fb4d, retrying]
May 11 11:15:53.069 - 303s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-11-082650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d50a615bc4172489a74c679b5e0894b7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f71fc915593dcd05a47c66c45de7fb4d, retrying]

... 1 lines not shown

May 11 05:13:33.502 E ns/e2e-check-for-dns-availability-9020 pod/dns-test-6a6b7197-5e53-4dd9-b3b9-ab11ab3940e0-dzrfx node/worker-0.ostest.test.metalkube.org uid/433e451a-ffef-4bdf-b6f2-ece5cae3e506 container/querier reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 11 05:14:19.413 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-11-022650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e0e317c3e285c4d1567f21fba513bfec expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ccbe0a6f4e53c6fb52f77cf9db59c1cd, retrying]
May 11 05:14:19.413 - 193s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-11-022650: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e0e317c3e285c4d1567f21fba513bfec expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ccbe0a6f4e53c6fb52f77cf9db59c1cd, retrying]

... 1 lines not shown

May 10 15:21:04.597 - 1s    E node/master-2.ostest.test.metalkube.org reason/FailedToDeleteCGroupsPath May 10 15:21:04.597325 master-2.ostest.test.metalkube.org hyperkube[3653]: I0510 15:21:04.597265    3653 pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods burstable podfe61852a5d4e5732d06cee70cc3fa85c] err="unable to destroy cgroup paths for cgroup [kubepods burstable podfe61852a5d4e5732d06cee70cc3fa85c] : Timed out while waiting for systemd to remove kubepods-burstable-podfe61852a5d4e5732d06cee70cc3fa85c.slice"
May 10 15:22:09.157 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-10-124007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5396c08514d2f658c3cde8ee062394b6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-96d988332b8e70f0de8b3a7c83cfde93, retrying]
May 10 15:22:09.157 - 240s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-10-124007: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5396c08514d2f658c3cde8ee062394b6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-96d988332b8e70f0de8b3a7c83cfde93, retrying]

... 1 lines not shown

May 10 08:11:42.163 E ns/openshift-kni-infra pod/haproxy-master-1.ostest.test.metalkube.org node/master-1.ostest.test.metalkube.org uid/1b4126e9-1ba0-4aad-b869-7fc07aaf4d75 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 08:12:08.200 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-10-052846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1989a432470163734e5be35fb04fb4c0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a31355f80c3a5fc84e059847a6156306, retrying]
May 10 08:12:08.200 - 655s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-10-052846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1989a432470163734e5be35fb04fb4c0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a31355f80c3a5fc84e059847a6156306, retrying]

... 1 lines not shown

May 14 16:07:37.020 - 999ms E disruption/oauth-api connection/reused reason/DisruptionBegan disruption/oauth-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ostest.test.metalkube.org:6443/apis/oauth.openshift.io/v1/oauthclients": net/http: timeout awaiting response headers
May 14 16:09:27.700 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-14-132144: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0228df85d05ea4f41ade0f3573680f39 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-eda57bfed489cc1dc6a938629c45df68, retrying]
May 14 16:09:27.700 - 220s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-14-132144: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0228df85d05ea4f41ade0f3573680f39 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-eda57bfed489cc1dc6a938629c45df68, retrying]

... 1 lines not shown

May 15 09:46:55.083 E ns/openshift-kni-infra pod/haproxy-master-1.ostest.test.metalkube.org node/master-1.ostest.test.metalkube.org uid/329d5e51-6259-463f-9105-cbda237e51e2 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 09:47:07.891 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-15-065158: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d5b5859a0cede49da4b51e6184c084fc expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b0dafac90fcb73925d39d4346f0ad33b, retrying]
May 15 09:47:07.891 - 703s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-15-065158: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d5b5859a0cede49da4b51e6184c084fc expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b0dafac90fcb73925d39d4346f0ad33b, retrying]

... 1 lines not shown

May 15 00:12:43.777 - 994ms E disruption/kube-api connection/reused reason/DisruptionBegan disruption/kube-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ostest.test.metalkube.org:6443/api/v1/namespaces/default": net/http: timeout awaiting response headers
May 15 00:13:41.707 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-14-213601: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4901373be75d30e7858ba508c75e2c28 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1d285d9d2bc6d0bac7d5656e7f601c85, retrying]
May 15 00:13:41.707 - 720s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-14-213601: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4901373be75d30e7858ba508c75e2c28 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1d285d9d2bc6d0bac7d5656e7f601c85, retrying]

... 1 lines not shown

May 10 02:14:12.855 E ns/openshift-kni-infra pod/haproxy-master-2.ostest.test.metalkube.org node/master-2.ostest.test.metalkube.org uid/0a220a92-62d6-454e-925b-0515ff7d4fb4 container/verify-api-int-resolvable reason/ContainerExit code/1 cause/Error
May 10 02:14:38.914 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-09-232846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c6588913b679850bb84c8130e1aa7f0b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-10b3d0cc1013389b7a13539e4b96be4e, retrying]
May 10 02:14:38.914 - 273s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-09-232846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c6588913b679850bb84c8130e1aa7f0b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-10b3d0cc1013389b7a13539e4b96be4e, retrying]

... 1 lines not shown

May 09 14:55:32.596 - 3s    E clusteroperator/network condition/Degraded status/True reason/Error while updating operator configuration: could not apply (network.operator.openshift.io/v1, Kind=OperatorPKI) openshift-ovn-kubernetes/signer: failed to apply / update (network.operator.openshift.io/v1, Kind=OperatorPKI) openshift-ovn-kubernetes/signer: Patch "https://api-int.ostest.test.metalkube.org:6443/apis/network.operator.openshift.io/v1/namespaces/openshift-ovn-kubernetes/operatorpkis/signer?fieldManager=cluster-network-operator%2Foperconfig&force=true": write tcp [fd2e:6f44:5dd8:c956::15]:38584->[fd2e:6f44:5dd8:c956::5]:6443: write: connection reset by peer
May 09 14:56:54.412 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d2f14528861de15f75c8e8a78d42c018 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-18928c1fd08d1d78d2ee2aab525d01e5, retrying]
May 09 14:56:54.412 - 217s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d2f14528861de15f75c8e8a78d42c018 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-18928c1fd08d1d78d2ee2aab525d01e5, retrying]

... 1 lines not shown

May 08 15:25:47.757 E ns/openshift-kni-infra pod/haproxy-master-0.ostest.test.metalkube.org node/master-0.ostest.test.metalkube.org uid/6408be2a-a2e6-44ab-ae63-255353f61966 container/verify-api-int-resolvable reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 15:26:11.207 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-08-124615: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9d517c95cdc0c228d2222d02bb4ca63a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-11d904ba88f23f880930ff4bd6984926, retrying]
May 08 15:26:11.207 - 689s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-08-124615: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9d517c95cdc0c228d2222d02bb4ca63a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-11d904ba88f23f880930ff4bd6984926, retrying]

... 1 lines not shown

May 08 05:54:39.062 - 999ms E disruption/kube-api connection/new reason/DisruptionBegan disruption/kube-api connection/new stopped responding to GET requests over new connections: error running request: 500 Internal Server Error: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Unknown desc = malformed header: missing HTTP content-type","code":500}\n
May 08 05:55:07.824 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-2024-05-08-030344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5bdcc1ee45a08e020507ab36ef961f5f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fb73567c00d166c24d07528ea53d4b16, retrying]
May 08 05:55:07.824 - 245s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-2024-05-08-030344: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5bdcc1ee45a08e020507ab36ef961f5f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fb73567c00d166c24d07528ea53d4b16, retrying]

... 1 lines not shown