May 15 20:57:58.814 E ns/e2e-k8s-sig-apps-job-upgrade-484 pod/foo-gsjql node/ip-10-0-162-84.us-east-2.compute.internal uid/0c1e05d4-6856-4eea-9024-f876bca8475b container/c reason/ContainerExit code/137 cause/Error
May 15 20:59:30.769 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-192157: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4e2be4cc2596793dc508ad4096a7bdb7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2839aa4a9c47d2cf310489084d069dc9, retrying]
May 15 20:59:30.769 - 484s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-192157: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4e2be4cc2596793dc508ad4096a7bdb7 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-2839aa4a9c47d2cf310489084d069dc9, retrying]

... 1 lines not shown

May 16 16:25:50.805 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-dcwcn node/ip-10-0-248-4.ec2.internal uid/045e72a1-50b2-49f0-8331-137022598531 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 16 16:27:14.810 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-16-144842: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2ecda19087ace20cdaa2852f7ba4e18 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-72f2aa75700288104ec0f9c9386354f6, retrying]
May 16 16:27:14.810 - 106s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-16-144842: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e2ecda19087ace20cdaa2852f7ba4e18 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-72f2aa75700288104ec0f9c9386354f6, retrying]

... 1 lines not shown

May 15 09:31:03.527 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-f6r4r node/ip-10-0-157-212.us-west-1.compute.internal uid/9d6883d1-bbea-44e5-8178-f19a47cf7fa6 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 15 09:32:17.331 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-075730: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-88ed16fdc18cc048c0672d2d4eb6ba16 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-cd06fdee62b8938bb445f969dc4b7173, retrying]
May 15 09:32:17.331 - 487s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-075730: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-88ed16fdc18cc048c0672d2d4eb6ba16 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-cd06fdee62b8938bb445f969dc4b7173, retrying]

... 1 lines not shown

May 15 17:37:06.584 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-c9b6b node/ip-10-0-239-180.us-east-2.compute.internal uid/097a6fe5-e038-457b-8155-c1b8de592ece container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 15 17:38:24.186 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-160117: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-01027f0a6af9bdf321a03f9e84f48f23 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-91e9de20a7aa2d8722a75fd8ad94c0d1, retrying]
May 15 17:38:24.186 - 474s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-160117: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-01027f0a6af9bdf321a03f9e84f48f23 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-91e9de20a7aa2d8722a75fd8ad94c0d1, retrying]

... 1 lines not shown

May 15 05:05:23.459 E ns/openshift-multus pod/multus-additional-cni-plugins-v9sq9 node/ip-10-0-240-106.ec2.internal uid/91c94a99-27fe-4025-81c8-41b3e789c0c3 container/kube-multus-additional-cni-plugins reason/TerminationStateCleared lastState.terminated was cleared on a pod (bug https://bugzilla.redhat.com/show_bug.cgi?id=1933760 or similar)
May 15 05:05:26.371 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-031851: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2c58b033c01c45f574fdd2638f4ca096 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7059f1db3429b57352186cd5da24999d, retrying]
May 15 05:05:26.371 - 446s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-031851: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-2c58b033c01c45f574fdd2638f4ca096 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7059f1db3429b57352186cd5da24999d, retrying]

... 1 lines not shown

May 11 13:29:04.692 E ns/openshift-apiserver pod/apiserver-7cd89bd8b8-9q7xw node/ip-10-0-203-158.us-west-1.compute.internal uid/731d7d95-ccd8-4abc-bd63-c9cc3bf3de5a container/openshift-apiserver-check-endpoints reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 11 13:31:59.016 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-11-115227: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7b9c40bc561b398758b63f30eb0c9de1 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b9f84ffc53c043835af5ad561af19c86, retrying]
May 11 13:31:59.016 - 96s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-11-115227: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7b9c40bc561b398758b63f30eb0c9de1 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-b9f84ffc53c043835af5ad561af19c86, retrying]

... 1 lines not shown

May 11 06:14:14.377 E ns/e2e-k8s-sig-apps-job-upgrade-7347 pod/foo-52599 node/ip-10-0-239-251.us-east-2.compute.internal uid/0e1008ab-f56a-4857-8ea3-338325138765 container/c reason/ContainerExit code/137 cause/Error
May 11 06:16:14.139 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-11-044154: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-947892f19e464cb484d27874635adbee expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6e76fc13ff04b4d8000326041614d96c, retrying]
May 11 06:16:14.139 - 465s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-11-044154: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-947892f19e464cb484d27874635adbee expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-6e76fc13ff04b4d8000326041614d96c, retrying]

... 1 lines not shown

May 14 23:12:35.637 - 1s    E ns/openshift-console route/console disruption/ingress-to-console connection/new reason/DisruptionBegan ns/openshift-console route/console disruption/ingress-to-console connection/new stopped responding to GET requests over new connections: Get "https://console-openshift-console.apps.ci-op-ympx943l-20826.aws-2.ci.openshift.org/healthz": read tcp 10.131.228.234:51710->44.239.72.29:443: read: connection reset by peer
May 14 23:13:27.565 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-14-213548: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c85e3bde7b20e67df5347dfc18b369de expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9112bd11b05c056bad6c4d0ffb92fa33, retrying]
May 14 23:13:27.565 - 97s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-14-213548: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c85e3bde7b20e67df5347dfc18b369de expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9112bd11b05c056bad6c4d0ffb92fa33, retrying]

... 1 lines not shown

May 15 02:00:09.038 E ns/e2e-k8s-sig-apps-job-upgrade-9200 pod/foo-k5fn7 node/ip-10-0-172-130.us-west-2.compute.internal uid/b9eb7ae8-3bb5-4861-ba84-7838a0dc7d3d container/c reason/ContainerExit code/137 cause/Error
May 15 02:02:25.265 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-002437: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-71921819e1f708c9b0bffa211cca521c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fd8659aef06679065a8457328df4c8c1, retrying]
May 15 02:02:25.265 - 464s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-15-002437: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-71921819e1f708c9b0bffa211cca521c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fd8659aef06679065a8457328df4c8c1, retrying]

... 1 lines not shown

May 14 14:57:20.597 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-skwwt node/ip-10-0-190-110.us-east-2.compute.internal uid/bd771c71-d600-4ffc-a09e-bbd1e9518a96 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 14 14:59:08.358 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-14-132146: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bad0444330abc108a165ce731d8b0c20 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9dac37392534838e6e981c3d045ed5d6, retrying]
May 14 14:59:08.358 - 109s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-14-132146: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bad0444330abc108a165ce731d8b0c20 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-9dac37392534838e6e981c3d045ed5d6, retrying]

... 1 lines not shown

May 11 03:33:41.751 - 58s   E alert/ClusterOperatorDown ns/openshift-cluster-version ALERTS{alertname="ClusterOperatorDown", alertstate="firing", name="machine-config", namespace="openshift-cluster-version", prometheus="openshift-monitoring/k8s", severity="critical"}
May 11 03:34:12.453 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-11-020012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-439dd3013a73e6a573f966d68d0f92d4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ff5fa48a5a931d49bc7fcca144524130, retrying]
May 11 03:34:12.453 - 100s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-11-020012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-439dd3013a73e6a573f966d68d0f92d4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ff5fa48a5a931d49bc7fcca144524130, retrying]

... 1 lines not shown

May 10 15:00:25.337 E ns/openshift-multus pod/multus-admission-controller-5fd5468f4f-s5msn node/ip-10-0-131-241.us-east-2.compute.internal uid/e6e3b39e-81c9-4f17-bd09-0ce614a64d18 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 15:03:58.987 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-10-081509: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c84b29bb6deed3b25958212681bb60ec expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-edd4711668f8aa67742892d0a31835c9, retrying]
May 10 15:03:58.987 - 109s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-10-081509: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c84b29bb6deed3b25958212681bb60ec expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-edd4711668f8aa67742892d0a31835c9, retrying]

... 1 lines not shown

May 10 09:44:22.002 E ns/e2e-k8s-sig-apps-job-upgrade-9887 pod/foo-hfbv5 node/ip-10-0-253-57.us-east-2.compute.internal uid/94a54059-9d51-4fa0-8e89-ee64d292239f container/c reason/ContainerExit code/137 cause/Error
May 10 09:46:20.913 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-10-081509: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c4759282135db186009de38144d8a718 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-dd3ba9c34c40d0fa5118c3f1b8aee605, retrying]
May 10 09:46:20.913 - 456s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-10-081509: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c4759282135db186009de38144d8a718 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-dd3ba9c34c40d0fa5118c3f1b8aee605, retrying]

... 1 lines not shown

May 10 15:38:24.183 E ns/e2e-k8s-sig-apps-job-upgrade-2429 pod/foo-g9466 node/ip-10-0-146-75.us-east-2.compute.internal uid/4043ecf6-8e94-4dc0-8e14-04bdb18d179e container/c reason/ContainerExit code/137 cause/Error
May 10 15:40:05.021 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-10-140740: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-70c2f76a417cf4111ad5feecbd45185f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d89a0a7128170bb76e93e4fc9f4243a6, retrying]
May 10 15:40:05.021 - 467s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-10-140740: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-70c2f76a417cf4111ad5feecbd45185f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-d89a0a7128170bb76e93e4fc9f4243a6, retrying]

... 1 lines not shown

May 10 04:00:41.602 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-165-20.us-east-2.compute.internal uid/c630d64b-1b56-41c9-b13f-af3640fc8155 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-10T03:35:10.009495457Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=rhaos-4.12-rhel-8, revision=a2a6d69)"\nlevel=info ts=2024-05-10T03:35:10.009545614Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20240509-20:14:59)"\nlevel=info ts=2024-05-10T03:35:10.009685528Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-10T03:35:14.29057647Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T03:35:14.290663724Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T03:35:19.678860342Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T03:36:41.750304366Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-10T03:37:53.787488373Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefil
May 10 04:01:13.923 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-10-023449: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-67c996e0f360733e904e0b8d2cdc26af expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a904839216fda97c8c9865dbe51bcd3c, retrying]
May 10 04:01:13.923 - 470s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-10-023449: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-67c996e0f360733e904e0b8d2cdc26af expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a904839216fda97c8c9865dbe51bcd3c, retrying]

... 1 lines not shown

May 10 01:04:48.687 E ns/openshift-multus pod/multus-admission-controller-5fd5468f4f-vw5jt node/ip-10-0-221-57.ec2.internal uid/960757c2-f589-4c7b-ae1f-895a15c1d46f container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 01:08:26.744 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-09-232822: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-82cfa93c61f3e34182e82e6d5f717626 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8c4d9193d9c6bbdaa2d4ae32047e9fc4, retrying]
May 10 01:08:26.744 - 85s   E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-09-232822: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-82cfa93c61f3e34182e82e6d5f717626 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-8c4d9193d9c6bbdaa2d4ae32047e9fc4, retrying]

... 1 lines not shown

May 09 13:42:54.872 E ns/e2e-k8s-sig-apps-job-upgrade-5947 pod/foo-pkp79 node/ip-10-0-184-126.us-east-2.compute.internal uid/9c355bcf-3ac4-4d2b-a79f-631a2b4719ea container/c reason/ContainerExit code/137 cause/Error
May 09 13:44:28.990 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-937c216346cbe293e22d37827b7c1436 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-68fd8c3bd1f66131b992da963c0c560f, retrying]
May 09 13:44:28.990 - 485s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-09-120424: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-937c216346cbe293e22d37827b7c1436 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-68fd8c3bd1f66131b992da963c0c560f, retrying]

... 1 lines not shown

May 08 14:14:24.884 E ns/e2e-k8s-sig-apps-job-upgrade-5118 pod/foo-pf72t node/ip-10-0-192-182.us-east-2.compute.internal uid/b95e1d1b-9491-40ca-8257-cd8f6816bf23 container/c reason/ContainerExit code/137 cause/Error
May 08 14:16:23.862 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-08-124412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ffd41a86ca44e5f685674af1c64cc2ca expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-43d00b0cc548e17081285321f23298a7, retrying]
May 08 14:16:23.862 - 471s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-08-124412: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-ffd41a86ca44e5f685674af1c64cc2ca expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-43d00b0cc548e17081285321f23298a7, retrying]

... 1 lines not shown

May 08 04:37:48.144 E ns/e2e-k8s-sig-apps-job-upgrade-5109 pod/foo-8sfvb node/ip-10-0-231-112.us-west-2.compute.internal uid/20c544b8-a355-4505-ac23-fd5309bdfb49 container/c reason/ContainerExit code/137 cause/Error
May 08 04:38:49.682 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.nightly-arm64-2024-05-08-030328: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7d660aab5a9c8747731123be88444b75 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-674ec8572c463e97e9da395194e4f9f4, retrying]
May 08 04:38:49.682 - 497s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.nightly-arm64-2024-05-08-030328: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7d660aab5a9c8747731123be88444b75 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-674ec8572c463e97e9da395194e4f9f4, retrying]

... 1 lines not shown