May 16 02:40:11.507 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-twmhzxvp-7f65d-bfst8-worker-centralus3-fk9nl uid/44236eee-5c17-4f83-95eb-c1ed6d8d5dfa container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-16T02:19:40.723712772Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-05-16T02:19:40.723789272Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-16T02:19:40.724085573Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-16T02:19:48.684859568Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-16T02:19:48.684993969Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-16T02:23:01.903910371Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\n
May 16 02:41:06.682 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-531ca0efd1e04923c4502e0b910feb4f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b53b6fb96f2b1121e65e86dd702afcfc, retrying]
May 16 02:41:06.682 - 419s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-531ca0efd1e04923c4502e0b910feb4f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-b53b6fb96f2b1121e65e86dd702afcfc, retrying]

... 1 lines not shown

May 15 17:40:43.937 E ns/openshift-multus pod/multus-additional-cni-plugins-bss5d node/ci-op-88z8nj0c-7f65d-7hvb7-worker-westus-g7mw5 uid/db75987a-5b98-412b-bcfb-c70c7de38b82 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 17:42:30.293 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-15-154441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cbf8f1bc914c4375423d3e4bd30d51c0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2bb19bd6c5bb753ff0fdac4985f91dc7, retrying]
May 15 17:42:30.293 - 91s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-15-154441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-cbf8f1bc914c4375423d3e4bd30d51c0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-2bb19bd6c5bb753ff0fdac4985f91dc7, retrying]

... 1 lines not shown

May 15 22:23:41.481 E ns/e2e-k8s-sig-apps-job-upgrade-4996 pod/foo-shzsh node/ci-op-yzstyx08-7f65d-btc8b-worker-eastus23-8x9cl uid/a86ba5e5-b600-419b-b3dd-3d3a1a6da6ef container/c reason/ContainerExit code/137 cause/Error
May 15 22:24:13.871 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-15-154441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2291d350ab74e90d46deb72d4ab55388 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-51b16725efb330c4b5beb63fb4946fbd, retrying]
May 15 22:24:13.871 - 441s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-15-154441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2291d350ab74e90d46deb72d4ab55388 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-51b16725efb330c4b5beb63fb4946fbd, retrying]

... 1 lines not shown

May 15 11:31:12.028 - 999ms E disruption/oauth-api connection/new reason/DisruptionBegan disruption/oauth-api connection/new stopped responding to GET requests over new connections: Get "https://api.ci-op-tkjdr8l4-7f65d.ci2.azure.devcluster.openshift.com:6443/apis/oauth.openshift.io/v1/oauthclients": net/http: timeout awaiting response headers
May 15 11:32:38.946 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-15-094441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-17fc0720345b6a8efa160b5e15f2aa4e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3dc5522fdf98503059bfa7ef274784f4, retrying]
May 15 11:32:38.946 - 89s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-15-094441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-17fc0720345b6a8efa160b5e15f2aa4e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-3dc5522fdf98503059bfa7ef274784f4, retrying]

... 1 lines not shown

May 15 02:57:50.042 - 2s    E disruption/kube-api connection/reused reason/DisruptionBegan disruption/kube-api connection/reused stopped responding to GET requests over reused connections: Get "https://api.ci-op-i440pgxy-7f65d.ci2.azure.devcluster.openshift.com:6443/api/v1/namespaces/default": net/http: timeout awaiting response headers
May 15 02:59:18.461 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-214846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5e56c7593273e4f5fe7eba147959edef expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-416de31dbcb7fbc23d7058fbdf1b3165, retrying]
May 15 02:59:18.461 - 95s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-214846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5e56c7593273e4f5fe7eba147959edef expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-416de31dbcb7fbc23d7058fbdf1b3165, retrying]

... 1 lines not shown

May 14 17:47:04.950 E ns/openshift-machine-config-operator pod/machine-config-server-pjqld node/ci-op-9y7j6blm-7f65d-b2vp4-master-1 uid/dbd97f1e-32c4-4ca0-9258-3483d489c5ef container/machine-config-server reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 14 17:47:15.932 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0f780608cfec734db7951597be117fe6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1bc4d800361e8a0cebd8761bcb7379d6, retrying]
May 14 17:47:15.932 - 366s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0f780608cfec734db7951597be117fe6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-1bc4d800361e8a0cebd8761bcb7379d6, retrying]

... 1 lines not shown

May 14 23:40:03.591 - 3s    E clusteroperator/authentication condition/Available status/False reason/OAuthServerDeploymentAvailable: deployment/openshift-authentication: could not be retrieved
May 14 23:40:54.669 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-214846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-de50bcf68a7e1aa50f1b442dca578f7e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-70e779fc990951922a1abfe565de4fff, retrying]
May 14 23:40:54.669 - 112s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-214846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-de50bcf68a7e1aa50f1b442dca578f7e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-70e779fc990951922a1abfe565de4fff, retrying]

... 1 lines not shown

May 14 03:01:41.410 E ns/e2e-k8s-sig-apps-job-upgrade-9176 pod/foo-4pxxr node/ci-op-wh97ycwi-7f65d-xh5wd-worker-eastus3-n2tdt uid/8754d82a-ddd6-47f9-8bea-02fa5c4e26ed container/c reason/ContainerExit code/137 cause/Error
May 14 03:01:46.218 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14d735c4a9feef6e00b1f08e95366f1f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f738c242196209ca0a27a34eca9de60f, retrying]
May 14 03:01:46.218 - 394s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-14d735c4a9feef6e00b1f08e95366f1f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f738c242196209ca0a27a34eca9de60f, retrying]

... 1 lines not shown

May 14 07:11:20.755 E ns/openshift-monitoring pod/kube-state-metrics-668b55746d-h74pp node/ci-op-vp687gs1-7f65d-sth8j-worker-centralus3-lqrp8 uid/9d583771-c04b-4759-a889-1c0b2bb243a9 container/kube-state-metrics reason/ContainerExit code/2 cause/Error
May 14 07:11:35.535 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-608e075b0656e2d80e8ce18c2cbd096e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3033180c5e0dc8ff9180992a9f5824c9, retrying]
May 14 07:11:35.535 - 441s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-608e075b0656e2d80e8ce18c2cbd096e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-3033180c5e0dc8ff9180992a9f5824c9, retrying]

... 1 lines not shown

May 13 23:49:10.958 - 55ms  E clusteroperator/storage condition/Degraded status/True reason/AzureFileCSIDriverOperatorDeploymentDegraded: client rate limiter Wait returned an error: context canceled
May 13 23:50:07.003 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d89ab59090c5651b3ba082649ace70ca expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8573e3882910dbf6000f136960c94611, retrying]
May 13 23:50:07.003 - 47s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d89ab59090c5651b3ba082649ace70ca expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8573e3882910dbf6000f136960c94611, retrying]

... 1 lines not shown

May 13 10:50:49.465 E ns/e2e-k8s-sig-apps-daemonset-upgrade-2347 pod/ds1-2jqdf node/ci-op-t1rrhch3-7f65d-njdkw-master-1 uid/c14d6005-5acf-4a47-b7d7-907b2ffd77bd container/app reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 13 10:50:50.069 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-090953: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c28456625a4c6bbd08004c6f48823f68 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-713d08e3b7be50bb9b91d2c09a71945b, retrying]
May 13 10:50:50.069 - 14s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-090953: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c28456625a4c6bbd08004c6f48823f68 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-713d08e3b7be50bb9b91d2c09a71945b, retrying]

... 1 lines not shown

May 11 23:14:32.051 E ns/openshift-dns pod/dns-default-r9vws node/ci-op-g8y1vgrd-7f65d-kmzk6-worker-westus-9kbzg uid/ec84f6bc-3ea7-41a2-b2a8-1020f15cccad container/kube-rbac-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 11 23:15:33.072 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e097d92536f2544bfbbe298ba7edb67d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-92f8fad72a9ee18d115369d45d8e4b65, retrying]
May 11 23:15:33.072 - 105s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e097d92536f2544bfbbe298ba7edb67d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-92f8fad72a9ee18d115369d45d8e4b65, retrying]

... 1 lines not shown

May 11 19:58:12.495 - 999ms E disruption/oauth-api connection/reused reason/DisruptionBegan disruption/oauth-api connection/reused stopped responding to GET requests over reused connections: error running request: 500 Internal Server Error: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Unknown desc = malformed header: missing HTTP content-type","code":500}\n
May 11 19:59:38.439 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7928d01f77ee0c937b98315e72155bab expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b0a0481f61e8ffe39b401f18fbb8010e, retrying]
May 11 19:59:38.439 - 66s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7928d01f77ee0c937b98315e72155bab expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b0a0481f61e8ffe39b401f18fbb8010e, retrying]

... 1 lines not shown

May 11 16:54:03.182 E ns/e2e-k8s-sig-apps-job-upgrade-5964 pod/foo-8szj4 node/ci-op-g7glc45z-7f65d-zl6gc-worker-eastus23-cjg4b uid/441e5736-8534-43bf-a135-1d97fe01fdef container/c reason/ContainerExit code/137 cause/Error
May 11 16:54:13.658 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d405227c073bb1ce07faab998d24c3d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-899f3b73a01edcfb1114bec0ce1c261d, retrying]
May 11 16:54:13.658 - 410s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d405227c073bb1ce07faab998d24c3d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-899f3b73a01edcfb1114bec0ce1c261d, retrying]

... 1 lines not shown

May 11 13:54:38.226 E ns/e2e-k8s-sig-apps-job-upgrade-529 pod/foo-n6jjj node/ci-op-b8wk4kcb-7f65d-ff22v-worker-centralus3-57tr8 uid/f4a19caf-2717-4947-a181-56fce5b67d8f container/c reason/ContainerExit code/137 cause/Error
May 11 13:54:56.969 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3f55ef772e1f7e0a6768d11c215a6a1b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fcf713149632649578b11bd3973c6df5, retrying]
May 11 13:54:56.969 - 428s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-3f55ef772e1f7e0a6768d11c215a6a1b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fcf713149632649578b11bd3973c6df5, retrying]

... 1 lines not shown

May 11 10:51:03.645 - 33s   E clusteroperator/network condition/Degraded status/True reason/Error while updating operator configuration: could not apply (network.openshift.io/v1, Kind=ClusterNetwork) /default: failed to apply / update (network.openshift.io/v1, Kind=ClusterNetwork) /default: etcdserver: request timed out, possibly due to previous leader failure
May 11 10:52:42.194 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d65ea56cd119639b822d28016a7b1aaa expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5cc4c8b6b30fdbf628ec808478670145, retrying]
May 11 10:52:42.194 - 67s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-d65ea56cd119639b822d28016a7b1aaa expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-5cc4c8b6b30fdbf628ec808478670145, retrying]

... 1 lines not shown

May 11 07:49:59.631 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-l2rhinms-7f65d-8lvxt-worker-eastus23-grp9h uid/8fe13ee3-a340-4904-821d-6f02ebfe3190 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-11T07:25:18.217919269Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-05-11T07:25:18.217995169Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-11T07:25:18.218224071Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-11T07:25:22.647224381Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-11T07:25:22.647349782Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-11T07:25:40.306864378Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-11T07:26:44.538478692Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-11T07:28:08.877830087Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles
May 11 07:49:59.915 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e46da66dd8d2ee8385b5c80b48fcd2df expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-608d255e14f6218401a102a3ebaea3ef, retrying]
May 11 07:49:59.915 - 391s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e46da66dd8d2ee8385b5c80b48fcd2df expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-608d255e14f6218401a102a3ebaea3ef, retrying]

... 1 lines not shown

May 11 03:36:16.904 E ns/openshift-multus pod/multus-additional-cni-plugins-vc4d6 node/ci-op-v5b1ycmr-7f65d-x69dp-master-1 uid/00ac4c1d-0ee6-440d-8e0d-26aac1e1fd57 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 11 03:36:23.897 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-df398344cdcbd8aa23351f1c912b1e7b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bdf3cfda7268b6da68c61790fc6fc829, retrying]
May 11 03:36:23.897 - 7s    E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-df398344cdcbd8aa23351f1c912b1e7b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-bdf3cfda7268b6da68c61790fc6fc829, retrying]

... 1 lines not shown

May 10 22:53:12.732 E ns/openshift-machine-config-operator pod/machine-config-daemon-4x29g node/ci-op-t9dk1y14-7f65d-jsf6b-master-0 uid/1672aec6-cb4a-4cc3-8943-1c10b62151ec container/oauth-proxy reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 10 22:53:23.728 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-132619: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-525428b1a3069a1739e88077e364907b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-475f0269b581cde5e33e0fb6e175a4c3, retrying]
May 10 22:53:23.728 - 438s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-132619: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-525428b1a3069a1739e88077e364907b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-475f0269b581cde5e33e0fb6e175a4c3, retrying]

... 1 lines not shown

May 10 18:05:52.375 E ns/openshift-monitoring pod/thanos-querier-fd4c66fb5-pnmrp node/ci-op-2sgfiqq0-7f65d-dts6g-worker-eastus23-6sn59 uid/b0e7c090-b796-4a85-b451-28c23ca53dec container/oauth-proxy reason/ContainerExit code/2 cause/Error 2024/05/10 17:56:25 provider.go:129: Defaulting client-id to system:serviceaccount:openshift-monitoring:thanos-querier\n2024/05/10 17:56:25 provider.go:134: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token\n2024/05/10 17:56:25 provider.go:358: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.\n2024/05/10 17:56:26 oauthproxy.go:203: mapping path "/" => upstream "http://localhost:9090/"\n2024/05/10 17:56:26 oauthproxy.go:224: compiled skip-auth-regex => "^/-/(healthy|ready)$"\n2024/05/10 17:56:26 oauthproxy.go:230: OAuthProxy configured for  Client ID: system:serviceaccount:openshift-monitoring:thanos-querier\n2024/05/10 17:56:26 oauthproxy.go:240: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain:<default> samesite: refresh:disabled\n2024/05/10 17:56:26 http.go:107: HTTPS: listening on [::]:9091\nI0510 17:56:26.015322       1 dynamic_serving_content.go:130] Starting serving::/etc/tls/private/tls.crt::/etc/tls/private/tls.key\n
May 10 18:06:13.700 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-132619: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f06e842f14e454ed7ddc1bd0c48aeb0e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-8150e5fa72efe1e47b041d2d8aa328b3, retrying]
May 10 18:06:13.700 - 426s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-132619: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f06e842f14e454ed7ddc1bd0c48aeb0e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-8150e5fa72efe1e47b041d2d8aa328b3, retrying]

... 1 lines not shown

May 10 15:13:28.342 - 24s   E clusteroperator/dns condition/Degraded status/True reason/DNS default is degraded
May 10 15:13:33.137 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-132619: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e3be4be9f05b45f2bc8efcec50ac652e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-af681137736f1ea2507bc02c654e5478, retrying]
May 10 15:13:33.137 - 89s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-132619: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e3be4be9f05b45f2bc8efcec50ac652e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-af681137736f1ea2507bc02c654e5478, retrying]

... 1 lines not shown

May 10 12:06:02.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://172.170.35.171:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 10 12:06:51.271 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e43489ce5bcbec6521084fd7a88dcf86 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-292cf936d3c7df205dc65c49bd347aa9, retrying]
May 10 12:06:51.271 - 48s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e43489ce5bcbec6521084fd7a88dcf86 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-292cf936d3c7df205dc65c49bd347aa9, retrying]

... 1 lines not shown

May 10 09:13:15.008 E ns/openshift-multus pod/multus-additional-cni-plugins-z6746 node/ci-op-b1y525g2-7f65d-grtqc-master-2 uid/dcd60f6f-ad1b-4a95-a2d1-492a1281faef container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 10 09:13:17.243 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0873aab4b51c041e85c19c7ae3d1cea4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-849a61f65c8c95974823365ed43c321b, retrying]
May 10 09:13:17.243 - 10s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-0873aab4b51c041e85c19c7ae3d1cea4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-849a61f65c8c95974823365ed43c321b, retrying]

... 1 lines not shown

May 10 06:10:13.090 E ns/openshift-cluster-csi-drivers pod/azure-disk-csi-driver-node-6tq5k node/ci-op-86y9hh8m-7f65d-tfjvn-master-2 uid/bce9a64b-ca0b-48f0-8a16-f326f6c0b54b container/csi-node-driver-registrar reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 10 06:10:13.159 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-982bec0942241c110a5ae6dd4882f5d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b9fd8fc90c5de97b52e5242968afac61, retrying]
May 10 06:10:13.159 - 41s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-982bec0942241c110a5ae6dd4882f5d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b9fd8fc90c5de97b52e5242968afac61, retrying]

... 1 lines not shown

May 09 20:16:08.091 E ns/openshift-sdn pod/sdn-jsxx6 node/ci-op-1kd29ks9-7f65d-bsp49-master-2 uid/39770307-e20b-4f54-b39e-14bc4dc4d2e1 container/drop-icmp reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 20:16:08.554 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7cdb0c01730092750222604b5af5a9f6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-c862dd4800cc3ed00228e11c60a129b0, retrying]
May 09 20:16:08.554 - 18s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7cdb0c01730092750222604b5af5a9f6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-c862dd4800cc3ed00228e11c60a129b0, retrying]

... 1 lines not shown

May 09 17:13:22.230 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-8zktsxvt-7f65d-lnn2j-worker-eastus23-nqx86 uid/90063512-ac2d-4351-bdb2-749d9621251e container/config-reloader reason/ContainerExit code/2 cause/Error 4.992556401Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T16:48:04.992672903Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T16:48:18.528446772Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T16:48:24.247074761Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T16:48:25.301742743Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T16:49:26.575327117Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-09T16:51:53.49813316Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/promethe
May 09 17:14:12.598 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-993dbfd2b56f312353326d0062fcc11c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2086b14654b432dcf1687a2045c4a5b7, retrying]
May 09 17:14:12.598 - 393s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-993dbfd2b56f312353326d0062fcc11c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2086b14654b432dcf1687a2045c4a5b7, retrying]

... 1 lines not shown

May 09 13:41:38.983 E ns/openshift-machine-config-operator pod/machine-config-daemon-rnnr2 node/ci-op-lsi7v62p-7f65d-2fvt4-master-1 uid/3598a8d7-49b1-4e8e-82bb-9c940fd3bea8 container/oauth-proxy reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 09 13:41:50.358 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-061655: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-30afebeca6c1d3782e20a4ee11063bb3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ab29eca2ec33bb724cd3161fa83a299f, retrying]
May 09 13:41:50.358 - 377s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-061655: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-30afebeca6c1d3782e20a4ee11063bb3 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-ab29eca2ec33bb724cd3161fa83a299f, retrying]

... 1 lines not shown

May 09 10:37:48.418 - 125ms E clusteroperator/authentication condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.user.openshift.io: not available: endpoints for service/api in "openshift-oauth-apiserver" have no addresses with port name "https"
May 09 10:38:22.226 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-061655: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-736a80f786ff901bcd3a8a93e5236a81 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-0232513a267e98466c77dd15d582745f, retrying]
May 09 10:38:22.226 - 509s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-061655: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-736a80f786ff901bcd3a8a93e5236a81 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-0232513a267e98466c77dd15d582745f, retrying]

... 1 lines not shown

May 09 01:53:46.201 E ns/openshift-e2e-loki pod/loki-promtail-fpbd2 node/ci-op-l0jq3wid-7f65d-zhrsr-worker-centralus3-f4b44 uid/09c0374e-cef2-4c09-b06b-7cb5d5e4d107 container/oauth-proxy reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 09 01:55:55.711 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-79cd425b80ab1c235ea0f73c7cf54e20 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-da29842266f9386ed8b6dc3ee34543bf, retrying]
May 09 01:55:55.711 - 53s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-79cd425b80ab1c235ea0f73c7cf54e20 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-da29842266f9386ed8b6dc3ee34543bf, retrying]

... 1 lines not shown

May 08 14:43:00.931 E ns/openshift-multus pod/multus-additional-cni-plugins-phx25 node/ci-op-7wv9njir-7f65d-2pb64-master-2 uid/3246be56-a1be-4c82-b6d4-73bb35924d97 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 08 14:43:05.464 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e23bacbf9b18c55d5ac6aa6bd722fb52 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b2b67c6cc16519fdd4ae598c678533b3, retrying]
May 08 14:43:05.464 - 7s    E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e23bacbf9b18c55d5ac6aa6bd722fb52 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b2b67c6cc16519fdd4ae598c678533b3, retrying]

... 1 lines not shown

May 08 07:51:02.059 E ns/openshift-multus pod/network-metrics-daemon-vchbx node/ci-op-8xglg1dj-7f65d-4kmnl-worker-centralus3-k8bqx uid/4032795d-3893-478f-b7c1-cdccb9a48dbf reason/ReusedPodIP podIP 10.131.0.4 is currently assigned to multiple pods: ns/openshift-e2e-loki pod/loki-promtail-xrt7l node/ci-op-8xglg1dj-7f65d-4kmnl-worker-centralus3-k8bqx uid/b1b68928-a19c-4fe0-867b-80b8ef87f82c;ns/openshift-multus pod/network-metrics-daemon-vchbx node/ci-op-8xglg1dj-7f65d-4kmnl-worker-centralus3-k8bqx uid/4032795d-3893-478f-b7c1-cdccb9a48dbf
May 08 07:53:01.528 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-023912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7b4504593289d685f0a97969936215d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-aeec78f0648ffeb78b6ad366abcab2a7, retrying]
May 08 07:53:01.528 - 49s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-023912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7b4504593289d685f0a97969936215d9 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-aeec78f0648ffeb78b6ad366abcab2a7, retrying]

... 1 lines not shown

May 08 04:25:23.998 E ns/e2e-k8s-sig-apps-job-upgrade-8113 pod/foo-v79dv node/ci-op-s0gmw0gc-7f65d-zfsgb-worker-centralus3-lgst5 uid/041c7f16-f079-4dc4-8627-5809b563845d container/c reason/ContainerExit code/137 cause/Error
May 08 04:25:52.537 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-023912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c06ce86c813a0c6963c9df5133e4c63f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c93af9b1409d4303c95d3041e7ddcb64, retrying]
May 08 04:25:52.537 - 434s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-023912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c06ce86c813a0c6963c9df5133e4c63f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-c93af9b1409d4303c95d3041e7ddcb64, retrying]

... 1 lines not shown

May 08 10:51:58.848 - 999ms E disruption/kube-api connection/new reason/DisruptionBegan disruption/kube-api connection/new stopped responding to GET requests over new connections: Get "https://api.ci-op-8b360kv7-7f65d.ci2.azure.devcluster.openshift.com:6443/api/v1/namespaces/default": net/http: timeout awaiting response headers
May 08 10:52:50.585 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-023912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e0d15427b64e17fb7e2e0b3be1ce3407 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-aac4261ac5391d7c7b65b01a5c961761, retrying]
May 08 10:52:50.585 - 120s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-023912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e0d15427b64e17fb7e2e0b3be1ce3407 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-aac4261ac5391d7c7b65b01a5c961761, retrying]

... 1 lines not shown

May 08 01:15:07.535 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-br8pv5v9-7f65d-x5fqr-worker-eastus3-ccmn9 uid/7bcdf313-d4b3-45de-99d5-2462473c2a72 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-08T00:44:01.57218762Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-05-08T00:44:01.572255826Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-08T00:44:01.572508049Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-08T00:44:07.857263799Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T00:44:07.85738761Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T00:44:22.659074031Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T00:45:48.553690978Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T00:52:20.139825229Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0
May 08 01:16:34.794 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-07-203912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e7e552317e2d9147a07655d8e433b47e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e189fbfd866e8cd5bc9595211f8e150c, retrying]
May 08 01:16:34.794 - 49s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-07-203912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e7e552317e2d9147a07655d8e433b47e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e189fbfd866e8cd5bc9595211f8e150c, retrying]

... 1 lines not shown

May 07 22:30:25.860 E ns/openshift-machine-config-operator pod/machine-config-server-vlgc7 node/ci-op-hqnl98jj-7f65d-d6w64-master-1 uid/c449d8fb-1732-4f60-b484-d31a527e8a4f container/machine-config-server reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 07 22:30:38.070 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-07-203912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a4e3385f762ec8c5cc0187aff7c00e82 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6d7a01d7f43113873de8cbb0a38cb7ad, retrying]
May 07 22:30:38.070 - 27s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-07-203912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a4e3385f762ec8c5cc0187aff7c00e82 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-6d7a01d7f43113873de8cbb0a38cb7ad, retrying]

... 1 lines not shown

May 07 18:06:58.611 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-qmtfb node/ci-op-xfdkswf4-7f65d-bvc5g-worker-eastus22-jrfmq uid/8d7fc1ed-3e28-470d-a3f4-1530e059dbeb container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 07 18:07:52.949 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-07-143912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bbe9fe2101c9c7547b774b5c4bddfbe1 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d11f0cc06f34a057d89fc09ee320af4c, retrying]
May 07 18:07:52.949 - 404s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-07-143912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bbe9fe2101c9c7547b774b5c4bddfbe1 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d11f0cc06f34a057d89fc09ee320af4c, retrying]

... 1 lines not shown

May 07 00:40:18.804 E ns/openshift-network-diagnostics pod/network-check-target-zbxws node/ci-op-fy2kvrx5-7f65d-wnvq8-worker-centralus3-58bvx uid/bdcbe235-3612-42db-8c97-504b2892757a container/network-check-target-container reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 07 00:41:16.550 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-05ce320e35623a61da8c96243992deb5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-300b360f21e46e36c67d6094ae71f92c, retrying]
May 07 00:41:16.550 - 87s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-05ce320e35623a61da8c96243992deb5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-300b360f21e46e36c67d6094ae71f92c, retrying]

... 1 lines not shown

May 06 14:34:13.407 E ns/openshift-monitoring pod/prometheus-k8s-1 node/ci-op-r6lyzgd9-7f65d-vw5zw-worker-westus-tw49s uid/d84d9313-dcfb-4f0a-bb20-a8fac2fe9584 container/config-reloader reason/ContainerExit code/2 cause/Error er=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-06T14:06:58.231970947Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-06T14:07:06.746031927Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-06T14:07:06.746182728Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-06T14:07:17.09651185Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-06T14:07:18.458216052Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-06T14:07:32.191945306Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-06T14:08:42.602363569Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/promethe
May 06 14:34:15.912 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-123542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c5770868ec7da6feb5feb2da6df1a67d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-244a8606abd94d650b5fad25f1ba5b43, retrying]
May 06 14:34:15.912 - 447s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-123542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c5770868ec7da6feb5feb2da6df1a67d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-244a8606abd94d650b5fad25f1ba5b43, retrying]

... 1 lines not shown

May 06 21:44:38.000 - 1s    E disruption/service-load-balancer-with-pdb connection/reused reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/reused stopped responding to GET requests over reused connections: Get "http://4.150.153.42:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 06 21:45:01.719 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-787ac595c925dc6161c916a118849693 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-7d4d3aaf43a590b7637411024c265ac4, retrying]
May 06 21:45:01.719 - 72s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-787ac595c925dc6161c916a118849693 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-7d4d3aaf43a590b7637411024c265ac4, retrying]

... 1 lines not shown

May 05 10:34:38.944 E ns/openshift-dns pod/dns-default-k6vkh node/ci-op-hxlh5k9s-7f65d-8qfgx-master-0 uid/e5a5151d-e3f0-49a1-bf26-69074c7e2a59 container/dns reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 05 10:34:49.000 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4c0491979e5faff8669bf0755602699d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-04bc125585e3bb4b622da3685a7e5acf, retrying]
May 05 10:34:49.000 - 30s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4c0491979e5faff8669bf0755602699d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-04bc125585e3bb4b622da3685a7e5acf, retrying]

... 1 lines not shown

May 04 04:38:42.389 E ns/e2e-k8s-sig-apps-job-upgrade-9258 pod/foo-jq24j node/ci-op-hwlm1k9f-7f65d-x8bgg-worker-eastus3-4zwl8 uid/faa3b9fe-0168-4002-9d81-95640c33964a container/c reason/ContainerExit code/137 cause/Error
May 04 04:39:16.063 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-35ee050d9e9194e26e505ba57edfd57a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7b9cea6014aaa419b89aade6d498e8f6, retrying]
May 04 04:39:16.063 - 388s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-35ee050d9e9194e26e505ba57edfd57a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7b9cea6014aaa419b89aade6d498e8f6, retrying]

... 1 lines not shown

May 04 01:36:50.525 - 135ms E clusteroperator/authentication condition/Available status/False reason/APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.user.openshift.io: not available: endpoints for service/api in "openshift-oauth-apiserver" have no addresses with port name "https"
May 04 01:38:50.462 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e711f9ba76ec8de35fbfe8425035fd17 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-94de1b38bc909f2d251a175e221bfee3, retrying]
May 04 01:38:50.462 - 44s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e711f9ba76ec8de35fbfe8425035fd17 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-94de1b38bc909f2d251a175e221bfee3, retrying]

... 1 lines not shown

May 03 22:39:46.350 - 114s  E clusteroperator/etcd condition/Degraded status/True reason/ClusterMemberControllerDegraded: unhealthy members found during reconciling members\nEtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:5343228253928061792 name:"ci-op-jt8201it-7f65d-dmpzh-master-0" peerURLs:"https://10.0.0.7:2380" clientURLs:"https://10.0.0.7:2379"  Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.0.7:2379]: context deadline exceeded} {Member:ID:12905433611601006678 name:"ci-op-jt8201it-7f65d-dmpzh-master-2" peerURLs:"https://10.0.0.8:2380" clientURLs:"https://10.0.0.8:2379"  Healthy:true Took:1.335232ms Error:<nil>} {Member:ID:13991265828320139868 name:"ci-op-jt8201it-7f65d-dmpzh-master-1" peerURLs:"https://10.0.0.6:2380" clientURLs:"https://10.0.0.6:2379"  Healthy:true Took:2.49466ms Error:<nil>}]\nEtcdMembersDegraded: 2 of 3 members are available, ci-op-jt8201it-7f65d-dmpzh-master-0 is unhealthy\nNodeControllerDegraded: The master nodes not ready: node "ci-op-jt8201it-7f65d-dmpzh-master-0" not ready since 2024-05-03 22:38:22 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
May 03 22:39:50.508 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c01c23a02c8e6c5ce907f796075936da expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-328b3f78f4c10ccf6c1df461f80eaa11, retrying]
May 03 22:39:50.508 - 123s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c01c23a02c8e6c5ce907f796075936da expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-328b3f78f4c10ccf6c1df461f80eaa11, retrying]

... 1 lines not shown

May 03 15:18:30.788 E ns/e2e-check-for-dns-availability-3659 pod/dns-test-b5737d97-5c8b-4291-a1cf-d0b42c0d9b2f-485cm node/ci-op-gqxnztmg-7f65d-2f6ss-worker-centralus3-vlpp5 uid/9b793997-4a28-4ad0-8806-46a50a705a5a container/querier reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 03 15:20:16.798 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-055af8a47f6abfbf3a26085a1f183cea expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-331fd80d04658fc5e24b3a7e1e529df3, retrying]
May 03 15:20:16.798 - 65s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-055af8a47f6abfbf3a26085a1f183cea expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-331fd80d04658fc5e24b3a7e1e529df3, retrying]

... 1 lines not shown

May 03 12:03:45.105 E ns/openshift-multus pod/multus-additional-cni-plugins-wkq9s node/ci-op-r31cjhfd-7f65d-zctsj-worker-centralus3-ndknl uid/6076c3b7-ac21-45b6-a498-263a5d64f5ad container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 03 12:04:26.233 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6c4334413a1e60cc556bf763af1e5a44 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-aa7ba1cc80e62cd2295f2d6ea91cd969, retrying]
May 03 12:04:26.233 - 93s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6c4334413a1e60cc556bf763af1e5a44 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-aa7ba1cc80e62cd2295f2d6ea91cd969, retrying]

... 1 lines not shown