May 16 21:12:09.410 E ns/openshift-multus pod/multus-additional-cni-plugins-86tkb node/ci-op-lksnn5cx-7cadb-sfl58-worker-eastus23-5j29p uid/956e699b-99d5-4610-b2ee-d2f186ab633f container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 21:14:20.959 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-16-170112: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8bc531ea05fb01f80ccd62a0029e9267 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0c89a9781c5d709d24d0eb66a1e888ff, retrying]
May 16 21:14:20.959 - 95s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-16-170112: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8bc531ea05fb01f80ccd62a0029e9267 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0c89a9781c5d709d24d0eb66a1e888ff, retrying]

... 1 lines not shown

May 15 17:21:39.750 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-2844c node/ci-op-mcxxitp2-7cadb-q6cnw-worker-eastus3-xbdwt uid/51433749-6a41-40aa-981d-fc5f847d5740 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 15 17:23:51.105 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-15-132734: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7e147e74e7ba0a1a725fa2dcba247690 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-c0ed17ce1fa01674a69eb7f918a9c481, retrying]
May 15 17:23:51.105 - 219s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-15-132734: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7e147e74e7ba0a1a725fa2dcba247690 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-c0ed17ce1fa01674a69eb7f918a9c481, retrying]

... 1 lines not shown

May 16 12:30:42.246 - 999ms E disruption/cache-oauth-api connection/reused reason/DisruptionBegan disruption/cache-oauth-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ci-op-qc2ccqnc-7cadb.ci2.azure.devcluster.openshift.com:6443/apis/oauth.openshift.io/v1/oauthclients?resourceVersion=0": net/http: timeout awaiting response headers
May 16 12:31:38.836 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-16-093540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9f313aa05e9287cd6c53a8cb70d8fc9d expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0ac2eb96be0d5007173539dacd4c051a, retrying]
May 16 12:31:38.836 - 147s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-16-093540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9f313aa05e9287cd6c53a8cb70d8fc9d expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-0ac2eb96be0d5007173539dacd4c051a, retrying]

... 1 lines not shown

May 16 15:35:26.872 E ns/openshift-multus pod/multus-additional-cni-plugins-7wnc5 node/ci-op-3i71mi1w-7cadb-7xxc6-worker-westus-lbgdd uid/2d169184-9cde-4f69-bb80-215ce45b97a2 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 16 15:35:48.218 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-16-093540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b1c3bb9be78a65bc7e7ec694a8d5034a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1d796d7e8d04dbc3a6dc230dfb6512a0, retrying]
May 16 15:35:48.218 - 797s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-16-093540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b1c3bb9be78a65bc7e7ec694a8d5034a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1d796d7e8d04dbc3a6dc230dfb6512a0, retrying]

... 1 lines not shown

May 16 05:52:13.641 - 999ms E disruption/oauth-api connection/reused reason/DisruptionBegan disruption/oauth-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ci-op-s1ginpt9-7cadb.ci2.azure.devcluster.openshift.com:6443/apis/oauth.openshift.io/v1/oauthclients": net/http: timeout awaiting response headers
May 16 05:54:49.546 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-16-033540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5bf8ee6d08b7df39a120685d442a647a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9ae45112f8237fead3c9a940a465facc, retrying]
May 16 05:54:49.546 - 132s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-16-033540: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5bf8ee6d08b7df39a120685d442a647a expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9ae45112f8237fead3c9a940a465facc, retrying]

... 1 lines not shown

May 14 23:38:12.186 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-pxc1qjz9-7cadb-zthx7-worker-centralus3-7d7q4 uid/66dd95ad-748a-4b3f-b7c2-d0679a04556a container/kube-rbac-proxy reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 14 23:39:50.141 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-14-214503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b231cfe93c3de69a8da8488ff9bd65c1 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5dfeae2adc7ac3999dbd8eda0063203b, retrying]
May 14 23:39:50.141 - 501s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-14-214503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b231cfe93c3de69a8da8488ff9bd65c1 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5dfeae2adc7ac3999dbd8eda0063203b, retrying]

... 1 lines not shown

May 15 05:41:14.047 E ns/openshift-cluster-csi-drivers pod/azure-disk-csi-driver-node-bgvm9 node/ci-op-vkqz2mjp-7cadb-8vzcf-worker-centralus2-g5vjg uid/22ca787d-77e8-4726-9ac0-122a339090cf container/csi-node-driver-registrar reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 05:41:14.475 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-15-034503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2d864b627d9109567b421db8c9dfcd0 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-895a0a887d3fed6b789bf23d8bd034da, retrying]
May 15 05:41:14.475 - 724s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-15-034503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2d864b627d9109567b421db8c9dfcd0 expected 2d7a09cc9acd5ffd96ef777a1bd2c348cd49a0c9 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-895a0a887d3fed6b789bf23d8bd034da, retrying]

... 1 lines not shown

May 14 17:35:57.014 E ns/e2e-k8s-sig-apps-job-upgrade-1275 pod/foo-lb4rr node/ci-op-l1qnkzi2-7cadb-l76h7-worker-centralus3-gb5f4 uid/1c38fafd-2df4-4aac-b995-def19d93e048 container/c reason/ContainerExit code/137 cause/Error
May 14 17:36:45.327 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-14-154503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5b2ac0c7b118e57e18eed3d2c3ab091c expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5f148661ce264d66cd3c111421defa2d, retrying]
May 14 17:36:45.327 - 504s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-14-154503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5b2ac0c7b118e57e18eed3d2c3ab091c expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5f148661ce264d66cd3c111421defa2d, retrying]

... 1 lines not shown

May 14 11:57:34.938 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-4n8ld node/ci-op-ssw1zbky-7cadb-rnt5q-worker-westus-jwjk9 uid/20a46301-341b-418a-9178-2007c2eb1c3e container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 14 11:59:50.348 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-14-094503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cc6cf5f45b994a13711cfc1cef5e1e9e expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fa8827ac292982477bf875a47a9989c3, retrying]
May 14 11:59:50.348 - 189s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-14-094503: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cc6cf5f45b994a13711cfc1cef5e1e9e expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fa8827ac292982477bf875a47a9989c3, retrying]

... 1 lines not shown

May 13 23:17:31.411 E ns/openshift-multus pod/multus-additional-cni-plugins-qcvjf node/ci-op-sbqksz94-7cadb-8qz6t-worker-centralus3-qx6d4 uid/2fed7be5-251f-4987-b6ad-5505eea81327 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 13 23:20:03.265 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-13-211327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e49c0901997ac937c528e125f3fd2dbe expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-22386149d6c4b944fec88772d4760e26, retrying]
May 13 23:20:03.265 - 130s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-13-211327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e49c0901997ac937c528e125f3fd2dbe expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-22386149d6c4b944fec88772d4760e26, retrying]

... 1 lines not shown

May 13 17:16:12.539 E ns/openshift-multus pod/multus-additional-cni-plugins-lfz5l node/ci-op-hixfv2qs-7cadb-qhhs4-worker-centralus3-tqhpv uid/748023c5-e9b1-4f38-a967-ce75d99778a8 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 13 17:18:07.754 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-13-151327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ee57507fdf02e5ae3e0550515379a6b2 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6a2c97b3dc6d0dbf7477f1a1adf9c365, retrying]
May 13 17:18:07.754 - 125s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-13-151327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ee57507fdf02e5ae3e0550515379a6b2 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6a2c97b3dc6d0dbf7477f1a1adf9c365, retrying]

... 1 lines not shown

May 13 11:20:43.742 E ns/e2e-k8s-sig-apps-job-upgrade-4577 pod/foo-jn6vr node/ci-op-9dhcc8xn-7cadb-z2rtb-worker-centralus3-6kkwb uid/630ef427-aeee-42b2-bce7-bfef7f8e6a11 container/c reason/ContainerExit code/137 cause/Error
May 13 11:20:48.515 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-13-091327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b7bf139c70229eb4fc750889ac343499 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-73e70d193cbe5615be50faa7cae41c3d, retrying]
May 13 11:20:48.515 - 517s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-13-091327: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b7bf139c70229eb4fc750889ac343499 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-73e70d193cbe5615be50faa7cae41c3d, retrying]

... 1 lines not shown

May 10 16:48:21.456 E ns/e2e-k8s-sig-apps-job-upgrade-4515 pod/foo-6jsmw node/ci-op-1qdzhj0m-7cadb-4jgdc-worker-eastus23-wmtsl uid/207f5d40-cfc7-4932-a648-7072b374a66f container/c reason/ContainerExit code/137 cause/Error
May 10 16:48:28.214 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-10-145431: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f820456f72afdf8477c5155bd1a61077 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-babb5b588def778746a7a00667f74b76, retrying]
May 10 16:48:28.214 - 451s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-10-145431: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f820456f72afdf8477c5155bd1a61077 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-babb5b588def778746a7a00667f74b76, retrying]

... 1 lines not shown

May 10 10:26:06.757 - 1s    E node/ci-op-9g3rcvcn-7cadb-vxckf-master-1 reason/FailedToDeleteCGroupsPath May 10 10:26:06.757681 ci-op-9g3rcvcn-7cadb-vxckf-master-1 kubenswrapper[1606]: I0510 10:26:06.757630    1606 pod_container_manager_linux.go:191] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pod8f91705a905e4c2f32a635d6e9bef1aa] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod8f91705a905e4c2f32a635d6e9bef1aa] : Timed out while waiting for systemd to remove kubepods-burstable-pod8f91705a905e4c2f32a635d6e9bef1aa.slice"
May 10 10:28:32.314 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-10-082301: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-46a04942c4c25165db6d741ea1dd1952 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ed6a69797563c2481aefa307da3e891b, retrying]
May 10 10:28:32.314 - 127s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-10-082301: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-46a04942c4c25165db6d741ea1dd1952 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ed6a69797563c2481aefa307da3e891b, retrying]

... 1 lines not shown

May 10 13:33:24.161 E ns/openshift-apiserver pod/apiserver-78d54995b5-n9slg node/ci-op-6h63fz2t-7cadb-bjkpd-master-0 uid/e19e2bc4-502d-4876-8cea-03739799c8a8 container/openshift-apiserver-check-endpoints reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 10 13:35:53.492 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-10-082301: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-88f29138105d6695dd2e8146cab4d3a6 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-67015da36d417b6fb1bee50b29e540b2, retrying]
May 10 13:35:53.492 - 114s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-10-082301: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-88f29138105d6695dd2e8146cab4d3a6 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-67015da36d417b6fb1bee50b29e540b2, retrying]

... 1 lines not shown

May 10 05:09:26.688 - 1s    E disruption/oauth-api connection/reused reason/DisruptionBegan disruption/oauth-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ci-op-bwnsc5np-7cadb.ci2.azure.devcluster.openshift.com:6443/apis/oauth.openshift.io/v1/oauthclients": net/http: timeout awaiting response headers
May 10 05:12:33.309 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-221809: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eb7b603c9792293a5781204f76452084 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-eb040ed6f6a1614bc74894e15ad1b883, retrying]
May 10 05:12:33.309 - 98s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-221809: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eb7b603c9792293a5781204f76452084 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-eb040ed6f6a1614bc74894e15ad1b883, retrying]

... 1 lines not shown

May 10 01:51:47.093 E ns/openshift-multus pod/multus-additional-cni-plugins-wfm6w node/ci-op-t34phyh4-7cadb-jxtfp-worker-westus-pwc6w uid/e7825026-c542-419f-9561-6ab927db1dc1 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 01:51:54.096 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-221809: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b2ec07474cddc63ff2723a4b93f629e6 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c503fc7e03168fbd4f7159fa81cfba98, retrying]
May 10 01:51:54.096 - 616s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-221809: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-b2ec07474cddc63ff2723a4b93f629e6 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c503fc7e03168fbd4f7159fa81cfba98, retrying]

... 1 lines not shown

May 09 20:58:16.590 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-xm0tds4n-7cadb-k5826-worker-eastus3-8bmtg uid/088b8c3b-4079-4487-b7d3-798af478bc7c container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-09T20:32:58.171925391Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.63.0, branch=release-4.13, revision=30fdccd13)"\nlevel=info ts=2024-05-09T20:32:58.172036993Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/amd64, user=root, date=20231108-08:14:29)"\nlevel=info ts=2024-05-09T20:32:58.172374601Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-09T20:33:05.210028576Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T20:33:05.210140678Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T20:33:18.395662149Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T20:34:45.181254519Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T20:40:29.908451334Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/pr
May 09 20:59:26.183 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-130510: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-549dc12943d48a781fae5c0166408708 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3c085dc69cc0fc463d7d5f564c4eb0df, retrying]
May 09 20:59:26.183 - 483s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-130510: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-549dc12943d48a781fae5c0166408708 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3c085dc69cc0fc463d7d5f564c4eb0df, retrying]

... 1 lines not shown

May 09 11:40:21.449 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-3g9rk104-7cadb-9x2mt-worker-centralus3-98bhq uid/0a670147-e606-4a5b-abc9-2107523c71e8 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-09T11:13:37.084676405Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.63.0, branch=release-4.13, revision=30fdccd13)"\nlevel=info ts=2024-05-09T11:13:37.084767408Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/amd64, user=root, date=20231108-08:14:29)"\nlevel=info ts=2024-05-09T11:13:37.085048219Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-09T11:13:44.066600996Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T11:13:44.0667261Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T11:13:52.397411535Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T11:15:20.706512109Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T11:22:49.543967683Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prom
May 09 11:40:52.504 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-004250: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0c16e76bbf7331611cc80971363a958c expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3d27e68d1ab3938d8c14f304ced2885b, retrying]
May 09 11:40:52.504 - 497s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-004250: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0c16e76bbf7331611cc80971363a958c expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3d27e68d1ab3938d8c14f304ced2885b, retrying]

... 1 lines not shown

May 09 05:58:12.161 E ns/openshift-multus pod/multus-additional-cni-plugins-6zzbf node/ci-op-iy4wcknh-7cadb-cb6tw-worker-westus-8qtm9 uid/77e342ed-2ebb-4e32-ba30-d99713678704 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 05:58:15.517 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-004250: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d0c6a4d7c1f5305ec4582307335751c5 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c1819d7bf2645811e90ac8565a4e092e, retrying]
May 09 05:58:15.517 - 639s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-004250: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d0c6a4d7c1f5305ec4582307335751c5 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c1819d7bf2645811e90ac8565a4e092e, retrying]

... 1 lines not shown

May 08 23:21:48.704 E ns/openshift-multus pod/multus-admission-controller-7fc4bb574f-pwhhg node/ci-op-mpdh7iw2-7cadb-5552m-master-1 uid/0e79d6d0-8df7-4108-aadc-f4e93e5c60e9 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 08 23:26:03.341 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-08-123136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5bc8df4aa392d5fbbacbfa3c5eb77924 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-28ef35c1670d16d7d2e2183869309599, retrying]
May 08 23:26:03.341 - 86s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-08-123136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5bc8df4aa392d5fbbacbfa3c5eb77924 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-28ef35c1670d16d7d2e2183869309599, retrying]

... 1 lines not shown

May 09 02:45:35.788 E ns/openshift-multus pod/multus-additional-cni-plugins-l9kxj node/ci-op-cwy69dnv-7cadb-xzrd6-worker-centralus3-zsp8l uid/4bc5b67d-51f1-4c5b-82e7-75ceee180c58 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 02:48:00.821 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-09-004250: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e90539fff956c4661fb69a09d5d783a0 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-93e8390f3dea437fae1bd5b08edd0c93, retrying]
May 09 02:48:00.821 - 105s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-09-004250: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e90539fff956c4661fb69a09d5d783a0 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-93e8390f3dea437fae1bd5b08edd0c93, retrying]

... 1 lines not shown

May 08 19:05:51.819 E ns/e2e-k8s-sig-apps-job-upgrade-9611 pod/foo-dnwnk node/ci-op-zmv97xl7-7cadb-fdw94-worker-eastus23-7cw77 uid/49fee259-f1b9-4a4a-9c33-f8844d35fa99 container/c reason/ContainerExit code/137 cause/Error
May 08 19:06:27.905 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-08-123136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-30f1bc164768c6a1d2a7704287535017 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-62a678e0d77bc6f0502b29aecbd139d2, retrying]
May 08 19:06:27.905 - 563s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-08-123136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-30f1bc164768c6a1d2a7704287535017 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-62a678e0d77bc6f0502b29aecbd139d2, retrying]

... 1 lines not shown

May 08 15:46:31.547 E ns/openshift-multus pod/multus-additional-cni-plugins-wgnrn node/ci-op-vn9ji9dr-7cadb-jmghr-worker-westus-ldtpc uid/c9eebe32-5db6-4d80-8366-38d627a8d42c container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 15:46:38.578 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-08-123136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5a7e744041b326dc3dce326afd6355c0 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2b55642e0e5b23f15985541d981543d2, retrying]
May 08 15:46:38.578 - 710s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-08-123136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5a7e744041b326dc3dce326afd6355c0 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2b55642e0e5b23f15985541d981543d2, retrying]

... 1 lines not shown

May 08 08:27:09.581 E ns/openshift-multus pod/multus-additional-cni-plugins-g24w6 node/ci-op-7kbwnhhd-7cadb-9f4zx-worker-westus-5dhrq uid/109b33e7-468c-4c30-86ab-f2f7b423fb15 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 08:27:12.404 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-08-063136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9afe689d7e2660235901909e10457dc1 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-27f092437a99d5d5d4ef65560ac3152a, retrying]
May 08 08:27:12.404 - 612s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-08-063136: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9afe689d7e2660235901909e10457dc1 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-27f092437a99d5d5d4ef65560ac3152a, retrying]

... 1 lines not shown

May 08 05:04:47.769 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-v4t8nwgn-7cadb-6dwfq-worker-centralus3-knzzv uid/cf32a53f-459c-41e5-b2c0-88ac4d29d5d2 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-08T04:39:17.106266424Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.63.0, branch=release-4.13, revision=30fdccd13)"\nlevel=info ts=2024-05-08T04:39:17.106382324Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, platform=linux/amd64, user=root, date=20231108-08:14:29)"\nlevel=info ts=2024-05-08T04:39:17.106656624Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=error ts=2024-05-08T04:39:17.110911427Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://localhost:9093/-/reload\": dial tcp [::1]:9093: connect: connection refused"\nlevel=info ts=2024-05-08T04:39:22.129144214Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/alertmanager/config/alertmanager.yaml.gz cfg_out=/etc/alertmanager/config_out/alertmanager.env.yaml watched_dirs="/etc/alertmanager/secrets/alertmanager-main-tls, /etc/alertmanager/secrets/alertmanager-main-proxy, /etc/alertmanager/secrets/alertmanager-kube-rbac-proxy, /etc/alertmanager/secrets/alertmanager-kube-rbac-proxy-metric"\nlevel=info ts=2024-05-08T04:39:22.129308214Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/alertmanager/config/alertmanager.yaml.gz out=/etc/alertmanager/config_out/alertmanager.env.yaml dirs=/etc/alertmanager/secrets/alertmanager-main-tls,/etc/alertmanager/secrets/alertmanager-main-proxy,/etc/alertmanager/secrets/alertmanager-kube-rbac-proxy,/etc/alertmanager/secrets/alertmanager-kube-rbac-proxy-metric\n
May 08 05:05:00.710 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-07-235844: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5ceef54001f0de05ab3ceedb6649ab2a expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f824f5c63edb91144ca3e7158cc3f3ad, retrying]
May 08 05:05:00.710 - 491s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-07-235844: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5ceef54001f0de05ab3ceedb6649ab2a expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f824f5c63edb91144ca3e7158cc3f3ad, retrying]

... 1 lines not shown

May 07 19:11:48.235 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-vx9tv6vg-7cadb-v94mr-worker-eastus22-28l7v uid/2259659a-8939-4084-ab5d-1c6e053ecab3 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-07T19:11:45.010Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=release-4.13, revision=df2f11e2ddc206ad1924ed766b4a394d84c9236f)"\nts=2024-05-07T19:11:45.010Z caller=main.go:241 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@50ec6f22b504, date=20231031-09:45:13)"\nts=2024-05-07T19:11:45.325Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-07T19:11:45.395Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-07T19:11:45.395Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-07T19:11:45.396Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=70.571503ms\n
May 07 19:12:27.141 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-07-154117: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-550f214571f8792d5eb65d26b1b7d7a4 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-13a2f4b599695a634697cd5e652f8525, retrying]
May 07 19:12:27.141 - 555s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-07-154117: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-550f214571f8792d5eb65d26b1b7d7a4 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-13a2f4b599695a634697cd5e652f8525, retrying]

... 1 lines not shown

May 07 14:09:55.872 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-d7qcp node/ci-op-2p9j2l9m-7cadb-jh2m2-worker-centralus1-bnw72 uid/7fc9b05a-9181-4803-9c9f-ff972bc2cb2b container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 07 14:13:17.254 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-07-085425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cfe5527105a5932965c2657ef9447224 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a1a22040817a0f85211a2d742752cea3, retrying]
May 07 14:13:17.254 - 111s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-07-085425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cfe5527105a5932965c2657ef9447224 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a1a22040817a0f85211a2d742752cea3, retrying]

... 1 lines not shown

May 07 10:59:06.857 - 999ms E disruption/cache-oauth-api connection/reused reason/DisruptionBegan disruption/cache-oauth-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ci-op-hbbfklsm-7cadb.ci2.azure.devcluster.openshift.com:6443/apis/oauth.openshift.io/v1/oauthclients?resourceVersion=0": net/http: timeout awaiting response headers
May 07 11:00:02.865 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-07-085425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-46ea9d31ae02cd897299e70bd11e41d8 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4d744859547a8ef9e4ef5e2a55c59131, retrying]
May 07 11:00:02.865 - 198s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-07-085425: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-46ea9d31ae02cd897299e70bd11e41d8 expected bb00d4f6f57ab3a3fc11644a958a6e1781389a23 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-4d744859547a8ef9e4ef5e2a55c59131, retrying]

... 1 lines not shown

May 06 22:07:10.631 E ns/openshift-multus pod/multus-additional-cni-plugins-fvptm node/ci-op-nq783mqy-7cadb-mwv7w-worker-centralus3-xm9bf uid/b1eca4af-b1d6-490d-9dc9-48e20dc9e863 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 06 22:08:30.973 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-06-192520: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4171bc7873fe0bebfe8d77100f215326 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2c668001a56bad0a59521df73dc03e8c, retrying]
May 06 22:08:30.973 - 138s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-06-192520: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4171bc7873fe0bebfe8d77100f215326 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2c668001a56bad0a59521df73dc03e8c, retrying]

... 1 lines not shown

May 06 18:07:44.738 E ns/openshift-multus pod/multus-additional-cni-plugins-nq7gg node/ci-op-ytxg7hk3-7cadb-zkts5-worker-centralus3-qzv7b uid/c5a3cb64-275e-4c34-b6bb-f2c6944c06b8 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 06 18:09:16.785 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-06-124130: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1b51c7ca07cedff3359eba1fd265cdf0 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-d5df3615f9a5d1c073104afd93ec575c, retrying]
May 06 18:09:16.785 - 114s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-06-124130: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1b51c7ca07cedff3359eba1fd265cdf0 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-d5df3615f9a5d1c073104afd93ec575c, retrying]

... 1 lines not shown

May 06 14:42:43.925 E ns/openshift-monitoring pod/thanos-querier-645df45b69-xhlf4 node/ci-op-1j8vfvyv-7cadb-ctnlf-worker-centralus3-7bkpl uid/d07931b3-92c0-4732-820f-a5fb375182b4 container/oauth-proxy reason/ContainerExit code/2 cause/Error 2024/05/06 14:16:35 provider.go:129: Defaulting client-id to system:serviceaccount:openshift-monitoring:thanos-querier\n2024/05/06 14:16:35 provider.go:134: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2024/05/06 14:16:35 provider.go:358: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2024/05/06 14:16:35 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:9090/"\n2024/05/06 14:16:35 oauthproxy.go:224: compiled skip-auth-regex => "^/-/(healthy|ready)$"\n2024/05/06 14:16:35 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:thanos-querier\n2024/05/06 14:16:35 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> samesite: refresh:disabled\nI0506 14:16:35.933995       1 dynamic_serving_content.go:130] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n2024/05/06 14:16:35 http.go:107: HTTPS: listening on [::]:9091\n
May 06 14:43:10.447 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-06-124130: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ed3428bd1819201431633f42d922db26 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-415f2d14c39b32f153a32cbec3bda926, retrying]
May 06 14:43:10.447 - 496s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-06-124130: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ed3428bd1819201431633f42d922db26 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-415f2d14c39b32f153a32cbec3bda926, retrying]

... 1 lines not shown

May 04 03:32:41.249 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-bfdq9 node/ci-op-qi4b6twi-7cadb-ln2f5-worker-eastus23-6x496 uid/d3e82090-5e9a-4001-bbfa-24fe9ec03749 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 04 03:35:36.797 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-03-162932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ae75e0be30a48cb4baf61fa9e70d1505 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-351cb2fffb9654a50dfd9c0817e0914e, retrying]
May 04 03:35:36.797 - 58s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-03-162932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ae75e0be30a48cb4baf61fa9e70d1505 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-351cb2fffb9654a50dfd9c0817e0914e, retrying]

... 1 lines not shown

May 03 03:12:08.135 - 1s    E node/ci-op-sl0lcim1-7cadb-4vv8p-master-1 reason/FailedToDeleteCGroupsPath May 03 03:12:08.135901 ci-op-sl0lcim1-7cadb-4vv8p-master-1 kubenswrapper[1606]: I0503 03:12:08.135846    1606 pod_container_manager_linux.go:191] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pode9fa0ce97e9a65d77e15ba70b4f32d3b] err="unable to destroy cgroup paths for cgroup [kubepods burstable pode9fa0ce97e9a65d77e15ba70b4f32d3b] : Timed out while waiting for systemd to remove kubepods-burstable-pode9fa0ce97e9a65d77e15ba70b4f32d3b.slice"
May 03 03:12:25.531 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-02-154457: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fcf144f0f40e30ff10295194c3b08e33 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6ccaeaf1eb635d883d393ed8622be498, retrying]
May 03 03:12:25.531 - 649s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-02-154457: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fcf144f0f40e30ff10295194c3b08e33 expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6ccaeaf1eb635d883d393ed8622be498, retrying]

... 1 lines not shown

May 04 00:19:50.252 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-9lx58 node/ci-op-wbcd4rq7-7cadb-hcttz-worker-centralus3-88zxb uid/a58aab8e-c221-4c39-bddb-354e07065329 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 04 00:22:14.764 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-03-162932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f83ee00caa736d800a16543db1393fec expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b5f3a8d20f9f4b54d66abd7d8fbbcb08, retrying]
May 04 00:22:14.764 - 168s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-03-162932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f83ee00caa736d800a16543db1393fec expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b5f3a8d20f9f4b54d66abd7d8fbbcb08, retrying]

... 1 lines not shown

May 03 20:28:25.940 E ns/e2e-k8s-sig-apps-job-upgrade-5845 pod/foo-tv2hd node/ci-op-jy0b70rd-7cadb-np2hs-worker-centralus3-2wpmj uid/11ef1440-e2e2-4b08-96a4-9aceb90b6e4a container/c reason/ContainerExit code/137 cause/Error
May 03 20:28:47.110 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-03-162932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-73728c24f4a5decbfe0be66663c1616c expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7ad90e13b92c2836173a7966158b2676, retrying]
May 03 20:28:47.110 - 498s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-03-162932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-73728c24f4a5decbfe0be66663c1616c expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7ad90e13b92c2836173a7966158b2676, retrying]

... 1 lines not shown

May 03 12:25:46.561 E ns/openshift-monitoring pod/alertmanager-main-0 node/ci-op-td38dq9n-7cadb-4p4kb-worker-centralus3-7tbbb uid/df470f11-f047-488b-9c1a-9cc7e0175c77 container/alertmanager reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 03 12:26:56.981 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.13.0-0.ci-2024-05-03-102932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bef6831b36c65f67c85b0498ff796fbf expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-bb6b363594a9d1b9e8b8001c31c3c43b, retrying]
May 03 12:26:56.981 - 457s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.13.0-0.ci-2024-05-03-102932: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bef6831b36c65f67c85b0498ff796fbf expected 88140227ee7d79c77092efd92b5a4363971333e2 has 33a010772d604aff2cb625d04a9469a47f53c96e: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-bb6b363594a9d1b9e8b8001c31c3c43b, retrying]

... 1 lines not shown