May 16 22:04:34.221 E ns/openshift-insights pod/insights-operator-6847d4ffc7-27tz9 node/master-2.ostest.test.metalkube.org uid/b3d26496-3130-4472-9364-7534a0414a08 container/insights-operator reason/ContainerExit code/2 cause/Error 9] k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:172: Watch close - *v1.ConfigMap total 4 items received\nI0516 22:03:10.938529       1 insightsuploader.go:122] Nothing to report since 0001-01-01T00:00:00Z\nI0516 22:03:14.180818       1 httplog.go:131] "HTTP" verb="GET" URI="/metrics" latency="2.810474ms" userAgent="Prometheus/2.42.0" audit-ID="f6f3afb4-88fc-462e-b6f0-364c5304595d" srcIP="[fd01:0:0:4::e]:53088" resp=200\nI0516 22:03:20.079849       1 httplog.go:131] "HTTP" verb="GET" URI="/metrics" latency="3.815819ms" userAgent="Prometheus/2.42.0" audit-ID="c70e607c-9242-4e1b-b681-4b318493e684" srcIP="[fd01:0:0:5::13]:51810" resp=200\nI0516 22:03:25.939642       1 insightsuploader.go:122] Nothing to report since 0001-01-01T00:00:00Z\nI0516 22:03:40.940066       1 insightsuploader.go:122] Nothing to report since 0001-01-01T00:00:00Z\nI0516 22:03:44.181293       1 httplog.go:131] "HTTP" verb="GET" URI="/metrics" latency="3.577638ms" userAgent="Prometheus/2.42.0" audit-ID="f8f188c7-6a44-4ed7-928f-46bf9398afd7" srcIP="[fd01:0:0:4::e]:53088" resp=200\nI0516 22:03:50.081015       1 httplog.go:131] "HTTP" verb="GET" URI="/metrics" latency="4.84678ms" userAgent="Prometheus/2.42.0" audit-ID="6511fd93-389e-4d1a-8a46-4026b6678602" srcIP="[fd01:0:0:5::13]:51810" resp=200\nI0516 22:03:55.940769       1 insightsuploader.go:122] Nothing to report since 0001-01-01T00:00:00Z\nI0516 22:04:10.941809       1 insightsuploader.go:122] Nothing to report since 0001-01-01T00:00:00Z\nI0516 22:04:14.226466       1 httplog.go:131] "HTTP" verb="GET" URI="/metrics" latency="6.22038ms" userAgent="Prometheus/2.42.0" audit-ID="235f8532-1f07-4379-ab60-0478f6445ec4" srcIP="[fd01:0:0:4::e]:53088" resp=200\nI0516 22:04:20.078964       1 httplog.go:131] "HTTP" verb="GET" URI="/metrics" latency="3.055781ms" userAgent="Prometheus/2.42.0" audit-ID="c51f06e1-2a3c-4ed4-b835-87d568513143" srcIP="[fd01:0:0:5::13]:51810" resp=200\nI0516 22:04:25.943022       1 insightsuploader.go:122] Nothing to report since 0001-01-01T00:00:00Z\n
May 16 22:04:34.375 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-16-191307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-93b0ab730c658f5f89c743c740271fac expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ee750d31605bc53f3bb5105a48fbd7ac, retrying]
May 16 22:04:34.375 - 278s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-16-191307: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-93b0ab730c658f5f89c743c740271fac expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ee750d31605bc53f3bb5105a48fbd7ac, retrying]

... 1 lines not shown

May 14 10:44:32.392 E ns/e2e-k8s-sig-apps-job-upgrade-2110 pod/foo-99jkk node/worker-1.ostest.test.metalkube.org uid/a0fc35cd-4ac1-4a98-9efd-128c9d71e435 container/c reason/ContainerExit code/137 cause/Error
May 14 10:44:33.390 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-14-080341: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ac7e13b55ffe45d669177aab98ebb7f1 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e91ac85cb541f22b31745cd57df250ce, retrying]
May 14 10:44:33.390 - 535s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-14-080341: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ac7e13b55ffe45d669177aab98ebb7f1 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-e91ac85cb541f22b31745cd57df250ce, retrying]

... 1 lines not shown

May 14 16:53:44.385 E ns/e2e-check-for-dns-availability-8983 pod/dns-test-9840778e-b7fc-48ea-b3a0-23193e7cd615-m6s2v node/worker-1.ostest.test.metalkube.org uid/ccfe0dd3-ced9-4855-b246-0ce5556d6d81 container/querier reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 14 16:56:16.698 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-14-140838: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3ff8d746093743b4fecbc54db7d398a8 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-35f3a934db4dae5c11b8dda3eb252c71, retrying]
May 14 16:56:16.698 - 114s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-14-140838: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3ff8d746093743b4fecbc54db7d398a8 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-35f3a934db4dae5c11b8dda3eb252c71, retrying]

... 1 lines not shown

May 10 16:33:03.687 E ns/openshift-dns pod/dns-default-f2c9l node/worker-2.ostest.test.metalkube.org uid/b2f393b9-4fb8-4947-a736-9bfa06d3b9fb container/dns reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 16:34:10.746 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-10-134150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cec8841f298059513f617ed8081062cf expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-de9f9bd35529af010122c3489ec8e1e4, retrying]
May 10 16:34:10.746 - 88s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-10-134150: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cec8841f298059513f617ed8081062cf expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-de9f9bd35529af010122c3489ec8e1e4, retrying]

... 1 lines not shown

May 10 07:36:38.306 E ns/openshift-kni-infra pod/haproxy-master-0.ostest.test.metalkube.org node/master-0.ostest.test.metalkube.org uid/884764bb-0214-4ad4-b32b-ea8bb410baad container/verify-api-int-resolvable reason/ContainerExit code/6 cause/Error
May 10 07:37:38.670 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-10-045757: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4567a16d04cc0155768ed9638f435504 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-57199a2555491d57de1078e66b318329, retrying]
May 10 07:37:38.670 - 647s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-10-045757: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4567a16d04cc0155768ed9638f435504 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-57199a2555491d57de1078e66b318329, retrying]

... 1 lines not shown

May 09 15:49:50.479 E ns/openshift-dns pod/dns-default-6tbrb node/worker-1.ostest.test.metalkube.org uid/002ce443-546f-4d88-8b21-82960786d540 container/kube-rbac-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 15:49:59.437 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-09-130302: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2de8e8a80b1f66a653ca98e40048d944 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3259015baab63bdc67f619e27abeca42, retrying]
May 09 15:49:59.437 - 550s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-09-130302: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2de8e8a80b1f66a653ca98e40048d944 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3259015baab63bdc67f619e27abeca42, retrying]

... 1 lines not shown

May 08 18:27:18.405 E ns/e2e-k8s-sig-apps-job-upgrade-9048 pod/foo-zrhlc node/worker-1.ostest.test.metalkube.org uid/1a618890-503b-4cc8-8b61-8dde807ced34 container/c reason/ContainerExit code/137 cause/Error
May 08 18:27:19.253 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-153833: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a4af75e44fa5ad5621d69f84fb3fc084 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-859badd224ef645a59db5a3bdac38c9f, retrying]
May 08 18:27:19.253 - 516s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-153833: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a4af75e44fa5ad5621d69f84fb3fc084 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-859badd224ef645a59db5a3bdac38c9f, retrying]

... 1 lines not shown

May 08 11:54:06.247 E ns/e2e-k8s-sig-apps-job-upgrade-459 pod/foo-l6z8f node/worker-1.ostest.test.metalkube.org uid/e1366e8f-0d6c-4653-9849-ac4313dcf2a2 container/c reason/ContainerExit code/137 cause/Error
May 08 11:54:18.836 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-090908: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-42fc765f4469855cb3e68b378560b11f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-112695b3691de08f05cd56de6a8dbd3d, retrying]
May 08 11:54:18.836 - 490s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-090908: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-42fc765f4469855cb3e68b378560b11f expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-112695b3691de08f05cd56de6a8dbd3d, retrying]

... 1 lines not shown

May 08 05:45:39.858 - 14s   E clusteroperator/dns condition/Degraded status/True reason/DNS default is degraded
May 08 05:46:21.050 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.nightly-2024-05-08-025210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-59fd978215cca68a0d1c96b10ed47fd0 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-61a1bc454c245d02f1637a95bee73fc3, retrying]
May 08 05:46:21.050 - 188s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.nightly-2024-05-08-025210: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-59fd978215cca68a0d1c96b10ed47fd0 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-61a1bc454c245d02f1637a95bee73fc3, retrying]

... 1 lines not shown