May 15 02:40:11.183 E ns/e2e-k8s-sig-apps-job-upgrade-1627 pod/foo-2hdlh node/ip-10-0-175-73.ec2.internal uid/c0cf10fa-478b-4abb-b9ae-2a9e5c3c99d8 container/c reason/ContainerExit code/137 cause/Error
May 15 02:41:03.747 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-214846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4d6c05c4e177bd3d19481590fe0daae6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-acf57f98bbdf921650b61d7b8d84aeac, retrying]
May 15 02:41:03.747 - 572s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-214846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4d6c05c4e177bd3d19481590fe0daae6 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-acf57f98bbdf921650b61d7b8d84aeac, retrying]

... 1 lines not shown

May 15 11:37:43.939 E ns/openshift-multus pod/multus-admission-controller-55d9fcf4f4-hjrxr node/ip-10-0-176-8.us-west-2.compute.internal uid/b92accd7-1f1e-4566-b548-4bd81b7b677a container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 15 11:41:05.953 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-15-094441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fdbac5ee30faac6319da2960119dc9e2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-218272c291e6f06996672c5353ee119d, retrying]
May 15 11:41:05.953 - 132s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-15-094441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-fdbac5ee30faac6319da2960119dc9e2 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-218272c291e6f06996672c5353ee119d, retrying]

... 1 lines not shown

May 14 23:33:27.171 E ns/openshift-monitoring pod/alertmanager-main-0 node/ip-10-0-143-35.us-west-2.compute.internal uid/88ae06b8-b0c6-4e92-acd9-7c47e73f8932 container/alertmanager reason/ContainerExit code/1 cause/Error ts=2024-05-14T23:33:23.953Z caller=main.go:231 level=info msg="Starting Alertmanager" version="(version=0.24.0, branch=release-4.12, revision=914cad827e9a177b29b23e02eb48b4065da8dca2)"\nts=2024-05-14T23:33:23.953Z caller=main.go:232 level=info build_context="(go=go1.19.13 X:strictfipsruntime, user=root@06d38c1ed95c, date=20231101-03:59:15)"\nts=2024-05-14T23:33:23.972Z caller=cluster.go:680 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s\nts=2024-05-14T23:33:23.996Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml\nts=2024-05-14T23:33:23.996Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/config_out/alertmanager.env.yaml err="open /etc/alertmanager/config_out/alertmanager.env.yaml: no such file or directory"\nts=2024-05-14T23:33:23.996Z caller=cluster.go:689 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=24.178894ms\n
May 14 23:34:35.048 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-214846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c0fb73768252b29d862bae80b2f5d25c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a96331f5a2e58a201cfbdead7b155b92, retrying]
May 14 23:34:35.048 - 559s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-214846: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c0fb73768252b29d862bae80b2f5d25c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-a96331f5a2e58a201cfbdead7b155b92, retrying]

... 1 lines not shown

May 15 17:45:01.154 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-vg42d node/ip-10-0-137-108.ec2.internal uid/c06e4167-87f0-4ea8-870d-6828324d0881 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 15 17:46:51.434 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-15-154441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c1401b8cecef927a113ccca6e6524552 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-38b5ddc4b9704387bba9a78e06c1c2b6, retrying]
May 15 17:46:51.434 - 149s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-15-154441: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-c1401b8cecef927a113ccca6e6524552 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-38b5ddc4b9704387bba9a78e06c1c2b6, retrying]

... 1 lines not shown

May 16 02:56:04.794 E ns/openshift-multus pod/multus-admission-controller-c6f4cd6f4-8xnv2 node/ip-10-0-194-175.us-west-1.compute.internal uid/8bdbacd0-469f-4a18-af71-7c83e36e3654 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 16 02:59:07.126 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-309858c2819c30721e46303f67681752 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-07786d8c741962bce70daefe0f7e85a3, retrying]
May 16 02:59:07.126 - 155s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-16-010831: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-309858c2819c30721e46303f67681752 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-07786d8c741962bce70daefe0f7e85a3, retrying]

... 1 lines not shown

May 14 17:34:34.577 E ns/e2e-k8s-sig-apps-job-upgrade-6179 pod/foo-qm9gc node/ip-10-0-159-239.ec2.internal uid/d427b710-9356-4244-b5e9-b4c5ce07e43e container/c reason/ContainerExit code/137 cause/Error
May 14 17:36:01.336 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8e6b3232d4c4d80a2246d402b2e56ff4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-317d63addbeeccda6a17a24823e8c80f, retrying]
May 14 17:36:01.336 - 522s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8e6b3232d4c4d80a2246d402b2e56ff4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-317d63addbeeccda6a17a24823e8c80f, retrying]

... 1 lines not shown

May 14 14:32:34.700 E ns/openshift-multus pod/multus-admission-controller-9b885445d-6j7b4 node/ip-10-0-243-165.ec2.internal uid/2cc43668-7a29-4beb-8fc0-6b05b334ae7b container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 14 14:35:43.418 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-689fe57581c13f78e917a898434d5c3f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fa23b1d1ba4cd95a1297d844182ad801, retrying]
May 14 14:35:43.418 - 149s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-689fe57581c13f78e917a898434d5c3f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-fa23b1d1ba4cd95a1297d844182ad801, retrying]

... 1 lines not shown

May 14 20:33:03.694 E ns/e2e-k8s-sig-apps-job-upgrade-2820 pod/foo-85f9s node/ip-10-0-179-85.ec2.internal uid/a8f2fce4-e2c9-4707-b514-18316af7969f container/c reason/ContainerExit code/137 cause/Error
May 14 20:34:12.795 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5e66e7f0d1156db9cb2f95f2db61345f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-314d4976f96eb1891962ca2d90537c06, retrying]
May 14 20:34:12.795 - 554s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-14-122945: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5e66e7f0d1156db9cb2f95f2db61345f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-314d4976f96eb1891962ca2d90537c06, retrying]

... 1 lines not shown

May 13 11:03:30.625 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-bv5q8 node/ip-10-0-180-123.ec2.internal uid/db667566-34b7-4545-83a9-330866a4e4a2 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 13 11:04:32.382 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-090953: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e7b98ba06f3cb87c7d0af5d15f40e6b8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-24499655485c91f3232deef4b0d22968, retrying]
May 13 11:04:32.382 - 162s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-090953: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-e7b98ba06f3cb87c7d0af5d15f40e6b8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-24499655485c91f3232deef4b0d22968, retrying]

... 1 lines not shown

May 12 19:22:39.090 E ns/openshift-multus pod/multus-admission-controller-58bd8d49f8-5z8zd node/ip-10-0-194-103.us-west-1.compute.internal uid/e4d43838-ed9b-47bc-ba51-ba752b648fd1 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 12 19:25:58.683 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dab6781201607530e0de2362c1c8d1da expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-143d02dfcc18e8ca2d62708eb11edbcc, retrying]
May 12 19:25:58.683 - 151s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-dab6781201607530e0de2362c1c8d1da expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-143d02dfcc18e8ca2d62708eb11edbcc, retrying]

... 1 lines not shown

May 11 20:00:29.460 E ns/e2e-k8s-sig-apps-job-upgrade-3109 pod/foo-krbbz node/ip-10-0-225-61.us-east-2.compute.internal uid/3c80b4b7-bca5-4c77-8ca6-8f969c346c49 container/c reason/ContainerExit code/137 cause/Error
May 11 20:01:18.119 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bf0f81dcb38f8982139aa15a1fb749f8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-34019b18e370568bf52ecece3ff89067, retrying]
May 11 20:01:18.119 - 189s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-181333: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bf0f81dcb38f8982139aa15a1fb749f8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-34019b18e370568bf52ecece3ff89067, retrying]

... 1 lines not shown

May 13 23:31:22.979 E ns/openshift-ingress pod/router-default-75bc45cf7b-f6wvf node/ip-10-0-193-7.ec2.internal uid/f83907f7-19e0-418d-859f-a3656b0d4ece container/router reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 13 23:32:32.962 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bbda043c2dbd9ff27fa1f29c2fc6e05c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-4f44e9e941340683f227052dc9d56670, retrying]
May 13 23:32:32.962 - 516s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-13-215426: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-bbda043c2dbd9ff27fa1f29c2fc6e05c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-4f44e9e941340683f227052dc9d56670, retrying]

... 1 lines not shown

May 10 15:16:40.815 E ns/openshift-multus pod/multus-admission-controller-547f7bfc8b-2l54j node/ip-10-0-244-162.us-west-1.compute.internal uid/8f2b9fce-c3f9-4a7c-926e-ccae9c0b1e03 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 15:20:05.241 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-132619: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-42bc07eb1146f51cd405212fb3f7d5ab expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f2bdd921406d08e5039015dd7aef009c, retrying]
May 10 15:20:05.241 - 234s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-132619: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-42bc07eb1146f51cd405212fb3f7d5ab expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f2bdd921406d08e5039015dd7aef009c, retrying]

... 1 lines not shown

May 11 01:49:26.482 E ns/e2e-k8s-sig-apps-job-upgrade-6609 pod/foo-d9dm8 node/ip-10-0-229-204.ec2.internal uid/fc9d065a-0064-434d-92fd-5a2adf900d9e container/c reason/ContainerExit code/137 cause/Error
May 11 01:50:25.562 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1109013efbd15d393200abbb04eaf8e8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-cb63f384f0f31f652b03483bed06cf9c, retrying]
May 11 01:50:25.562 - 563s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-001012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1109013efbd15d393200abbb04eaf8e8 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-cb63f384f0f31f652b03483bed06cf9c, retrying]

... 1 lines not shown

May 11 08:05:22.863 E ns/openshift-cloud-credential-operator pod/pod-identity-webhook-8b78fb796-qg5dw node/ip-10-0-179-122.us-east-2.compute.internal uid/9ba8dbec-18c4-454c-bc20-dc6a346ece39 container/pod-identity-webhook reason/ContainerExit code/137 cause/Error
May 11 08:08:32.806 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-84a8ce2b8824d877cc2931f367431bf4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a15e4d9a95d51aed33be02b3636cc97a, retrying]
May 11 08:08:32.806 - 150s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-11-061012: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-84a8ce2b8824d877cc2931f367431bf4 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a15e4d9a95d51aed33be02b3636cc97a, retrying]

... 1 lines not shown

May 10 09:17:25.563 E ns/e2e-k8s-sig-apps-job-upgrade-1278 pod/foo-vtnjn node/ip-10-0-135-143.us-east-2.compute.internal uid/2dff5e7b-c40a-459b-9a3c-e0a4d5b281b8 container/c reason/ContainerExit code/137 cause/Error
May 10 09:18:31.429 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5e6f512692ce2a6567509e88808de411 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7dbea730fba044c5e83d3692d05f2fb6, retrying]
May 10 09:18:31.429 - 550s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-5e6f512692ce2a6567509e88808de411 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7dbea730fba044c5e83d3692d05f2fb6, retrying]

... 1 lines not shown

May 10 06:20:42.227 E ns/openshift-multus pod/multus-admission-controller-597668d658-lqfrf node/ip-10-0-169-82.us-west-2.compute.internal uid/8f2aed2f-f31f-4107-888d-70fe2ad2c5f2 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 10 06:23:34.092 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4e2dc7eb5ed262d54f76e4810339dd9f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-029a3728360243421e6fd7b6a12e764e, retrying]
May 10 06:23:34.092 - 164s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-10-043105: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4e2dc7eb5ed262d54f76e4810339dd9f expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-029a3728360243421e6fd7b6a12e764e, retrying]

... 1 lines not shown

May 10 03:16:33.297 E ns/e2e-k8s-sig-apps-job-upgrade-7720 pod/foo-wtm5k node/ip-10-0-244-130.ec2.internal uid/b6c3ba0f-4f3c-4c08-9de5-488944db60da container/c reason/ContainerExit code/137 cause/Error
May 10 03:17:49.366 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-62b3b2efc809f212dbf636ce430ac59d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fccca0266e3d32f0829cb82a3b82f1b4, retrying]
May 10 03:17:49.366 - 514s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-62b3b2efc809f212dbf636ce430ac59d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fccca0266e3d32f0829cb82a3b82f1b4, retrying]

... 1 lines not shown

May 09 23:37:47.563 - 999ms E ns/openshift-console route/console disruption/ingress-to-console connection/new reason/DisruptionBegan ns/openshift-console route/console disruption/ingress-to-console connection/new stopped responding to GET requests over new connections: Get "https://console-openshift-console.apps.ci-op-zvl6bgn6-d5c74.aws-2.ci.openshift.org/healthz": read tcp 10.130.56.101:34190->44.198.218.194:443: read: connection reset by peer
May 09 23:38:17.504 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-504fb8058bd7c3fe28ab08e4952239ca expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5957e78ebce55a973f50509871c9adf3, retrying]
May 09 23:38:17.504 - 514s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-504fb8058bd7c3fe28ab08e4952239ca expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-5957e78ebce55a973f50509871c9adf3, retrying]

... 1 lines not shown

May 09 08:15:27.851 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-zjx6w node/ip-10-0-148-53.us-west-2.compute.internal uid/5c288cb0-d35b-48b4-ba4f-c06119d9f8d5 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 09 08:16:10.830 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-061655: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9523fcc9c10c894b69054c46bc1e6e61 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-46d64564132d70163b0f67b142b6a11e, retrying]
May 09 08:16:10.830 - 558s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-061655: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9523fcc9c10c894b69054c46bc1e6e61 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-46d64564132d70163b0f67b142b6a11e, retrying]

... 1 lines not shown

May 09 03:39:52.770 E ns/e2e-k8s-sig-apps-job-upgrade-6381 pod/foo-vrzxx node/ip-10-0-184-132.ec2.internal uid/ff0bd157-5277-4c48-b258-6ab101e4257e container/c reason/ContainerExit code/137 cause/Error
May 09 03:41:12.614 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9c002b88d7e44da69c318a0bf74b31ff expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fb1a89e973ef8f88255ce459bd401606, retrying]
May 09 03:41:12.614 - 569s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-9c002b88d7e44da69c318a0bf74b31ff expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-fb1a89e973ef8f88255ce459bd401606, retrying]

... 1 lines not shown

May 09 17:01:05.890 E ns/openshift-multus pod/multus-admission-controller-568846ff78-pxtr5 node/ip-10-0-148-38.ec2.internal uid/414f3853-2fd6-4140-8155-fbbcd7fa3e1b container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 09 17:04:13.208 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-83c04dae506f3db1db013ff68d412e66 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a0baf8dc8053087de44a683f3e8ccdfc, retrying]
May 09 17:04:13.208 - 151s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-09-151252: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-83c04dae506f3db1db013ff68d412e66 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-a0baf8dc8053087de44a683f3e8ccdfc, retrying]

... 1 lines not shown

May 09 00:44:56.296 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-fshq8 node/ip-10-0-176-59.us-east-2.compute.internal uid/f1d6d98b-2e91-4b24-b868-7316fd118885 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 09 00:47:14.294 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eae808eb7a0679fdefdd9d29d02e2967 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ae2fbc1c294c6ebee38644ec5be5052b, retrying]
May 09 00:47:14.294 - 143s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-eae808eb7a0679fdefdd9d29d02e2967 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-ae2fbc1c294c6ebee38644ec5be5052b, retrying]

... 1 lines not shown

May 08 21:34:33.069 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-156-37.us-west-1.compute.internal uid/39e61185-813d-40d4-959f-15f5bc139641 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-08T21:00:23.110321786Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-05-08T21:00:23.110402987Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-08T21:00:23.11060347Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-08T21:00:27.521129371Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T21:00:27.521232283Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T21:00:41.64175086Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T21:03:22.56566116Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T21:08:14.588501776Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\n
May 08 21:35:00.428 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8cbc6db76fd6687c7027aa9ecb1a293c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7b90bb1b33a9e001f679af4bc8078b54, retrying]
May 08 21:35:00.428 - 556s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-194829: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8cbc6db76fd6687c7027aa9ecb1a293c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-7b90bb1b33a9e001f679af4bc8078b54, retrying]

... 1 lines not shown

May 08 04:27:42.479 - 131ms E clusteroperator/storage condition/Degraded status/True reason/AWSEBSCSIDriverOperatorDeploymentDegraded: Get "https://172.30.0.1:443/apis/apps/v1/namespaces/openshift-cluster-csi-drivers/deployments/aws-ebs-csi-driver-operator": context canceled
May 08 04:27:59.947 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-023912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-10fe76f84e0f47623bcd1ae8108abb06 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f353070a16e7cdb5182faf7b7f63daa1, retrying]
May 08 04:27:59.947 - 156s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-023912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-10fe76f84e0f47623bcd1ae8108abb06 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-f353070a16e7cdb5182faf7b7f63daa1, retrying]

... 1 lines not shown

May 07 22:34:29.444 - 1s    E node/ip-10-0-150-123.us-west-1.compute.internal reason/FailedToDeleteCGroupsPath May 07 22:34:29.444189 ip-10-0-150-123 hyperkube[1959]: I0507 22:34:29.444073    1959 pod_container_manager_linux.go:192] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pod51ea591b-bf4e-4e68-911b-4619837ef230] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod51ea591b-bf4e-4e68-911b-4619837ef230] : Timed out while waiting for systemd to remove kubepods-burstable-pod51ea591b_bf4e_4e68_911b_4619837ef230.slice"
May 07 22:37:26.479 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-07-203912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4b17b0af4dfb02e5574b13a2a2820d0c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-10556c6677c548af7a0b97882d1fc0e9, retrying]
May 07 22:37:26.479 - 166s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-07-203912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-4b17b0af4dfb02e5574b13a2a2820d0c expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-10556c6677c548af7a0b97882d1fc0e9, retrying]

... 1 lines not shown

May 08 14:43:02.591 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-185-115.ec2.internal uid/30b87981-4b89-476a-865b-57773941bbb9 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-08T14:08:07.380434768Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-05-08T14:08:07.38050577Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-08T14:08:07.380681293Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-08T14:08:12.209857477Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T14:08:12.209955019Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T14:10:41.189068377Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-08T14:16:41.11309913Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\n
May 08 14:44:08.182 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-44459add6f0fe8648a0c12b6aef5ba6e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-0366554f851cdd7c851be66d71bb4fe0, retrying]
May 08 14:44:08.182 - 554s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-08-124617: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-44459add6f0fe8648a0c12b6aef5ba6e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-0366554f851cdd7c851be66d71bb4fe0, retrying]

... 1 lines not shown

May 07 16:32:02.050 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-d8cbk node/ip-10-0-168-55.us-west-1.compute.internal uid/52d2617f-3c35-4f78-882b-d9f07c3fe177 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 07 16:34:10.454 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-07-143912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f7aff296b46415a0d3c6cd0f64af9a57 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-43beadf2dbac7cddbbed0087f5929363, retrying]
May 07 16:34:10.454 - 161s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-07-143912: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f7aff296b46415a0d3c6cd0f64af9a57 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-43beadf2dbac7cddbbed0087f5929363, retrying]

... 1 lines not shown

May 06 20:27:53.305 E ns/openshift-multus pod/multus-admission-controller-68fb944f46-9h6rj node/ip-10-0-207-251.ec2.internal uid/5ecebb8e-5cb3-44a8-b613-c14539ba66c6 container/multus-admission-controller reason/ContainerExit code/137 cause/Error
May 06 20:31:16.997 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-437e849f52ad1cb66e3a048df053254e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-cab14fc6499352632228e104ee4fe968, retrying]
May 06 20:31:16.997 - 123s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-183542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-437e849f52ad1cb66e3a048df053254e expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-cab14fc6499352632228e104ee4fe968, retrying]

... 1 lines not shown

May 06 14:31:33.202 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-4xzfm node/ip-10-0-178-22.us-west-1.compute.internal uid/0260d9b8-dd9f-4191-b121-d0ccbd104e62 container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 06 14:32:54.634 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-06-123542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1ee9480b6e1dcb29b4068aef5357e49d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-32f4ed3cfae49d96b1ae192c8973df45, retrying]
May 06 14:32:54.634 - 168s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-06-123542: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-1ee9480b6e1dcb29b4068aef5357e49d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-32f4ed3cfae49d96b1ae192c8973df45, retrying]

... 1 lines not shown

May 04 03:30:38.399 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-152-89.ec2.internal uid/0ac0756a-adf5-4d45-87d0-9e3ffac303b3 container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-04T02:55:23.223003923Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-05-04T02:55:23.223073855Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-04T02:55:23.223257859Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-04T02:55:27.069652916Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-04T02:55:27.069749428Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-04T02:57:04.18908235Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-04T03:03:06.240114334Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\n
May 04 03:31:20.387 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-12f9e0a9b94ba4f74e2344c64046272b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-103fa3ff859a884e13b2af404f8b74e9, retrying]
May 04 03:31:20.387 - 513s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-12f9e0a9b94ba4f74e2344c64046272b expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-103fa3ff859a884e13b2af404f8b74e9, retrying]

... 1 lines not shown

May 04 00:22:35.291 E ns/openshift-monitoring pod/prometheus-k8s-0 node/ip-10-0-160-102.ec2.internal uid/b2842346-51bd-4aee-9b46-c3f4e06a55ce container/config-reloader reason/ContainerExit code/2 cause/Error level=info ts=2024-05-03T23:50:46.638911769Z caller=main.go:111 msg="Starting prometheus-config-reloader" version="(version=0.60.1, branch=release-4.12, revision=d1e399d5c)"\nlevel=info ts=2024-05-03T23:50:46.63897296Z caller=main.go:112 build_context="(go=go1.19.13 X:strictfipsruntime, user=root, date=20231109-13:38:17)"\nlevel=info ts=2024-05-03T23:50:46.639114442Z caller=main.go:149 msg="Starting web server for metrics" listen=localhost:8080\nlevel=info ts=2024-05-03T23:50:51.037993749Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-03T23:50:51.0380949Z caller=reloader.go:235 msg="started watching config file and directories for changes" cfg=/etc/prometheus/config/prometheus.yaml.gz out=/etc/prometheus/config_out/prometheus.env.yaml dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-03T23:53:52.054020056Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\nlevel=info ts=2024-05-03T23:57:46.193167207Z caller=reloader.go:374 msg="Reload triggered" cfg_in=/etc/prometheus/config/prometheus.yaml.gz cfg_out=/etc/prometheus/config_out/prometheus.env.yaml watched_dirs=/etc/prometheus/rules/prometheus-k8s-rulefiles-0\n
May 04 00:24:55.274 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a3d6ce7c7d012817c93da2a373489a23 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f104eb1e2706d32bebb06fe369ee8380, retrying]
May 04 00:24:55.274 - 530s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-a3d6ce7c7d012817c93da2a373489a23 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f104eb1e2706d32bebb06fe369ee8380, retrying]

... 1 lines not shown

May 03 18:14:39.093 E ns/openshift-multus pod/multus-admission-controller-fc74654bb-wbgtq node/ip-10-0-146-78.us-east-2.compute.internal uid/d1c0a234-9a6e-484c-83c9-67b9273e91f4 container/multus-admission-controller reason/ContainerExit code/137 cause/ContainerStatusUnknown The container could not be located when the pod was deleted.  The container used to be Running
May 03 18:17:38.695 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-000aeb57e73a00de22ff91c19ac1eed5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-75e5270f825e9b94438cce8a7cf92f44, retrying]
May 03 18:17:38.695 - 147s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-000aeb57e73a00de22ff91c19ac1eed5 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-75e5270f825e9b94438cce8a7cf92f44, retrying]

... 1 lines not shown

May 03 15:13:38.000 - 36s   E disruption/service-load-balancer-with-pdb connection/new reason/DisruptionBegan disruption/service-load-balancer-with-pdb connection/new stopped responding to GET requests over new connections: Get "http://a8414a863a3154fd191a3c73d11c4bab-184536117.us-west-2.elb.amazonaws.com:80/echo?msg=Hello": net/http: timeout awaiting response headers
May 03 15:13:56.939 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-73ede303cf87313e92cb85b7abfc648a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-549fbf8b0785107827234260557caade, retrying]
May 03 15:13:56.939 - 539s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-73ede303cf87313e92cb85b7abfc648a expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-549fbf8b0785107827234260557caade, retrying]

... 1 lines not shown

May 03 21:23:29.326 E ns/openshift-multus pod/cni-sysctl-allowlist-ds-zk5r7 node/ip-10-0-204-164.us-west-1.compute.internal uid/39b113d9-74ca-4586-9d2b-562102ff81cb container/kube-multus-additional-cni-plugins reason/ContainerExit code/137 cause/Error
May 03 21:25:25.099 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8eb304e9ec5b058bbebd5297505e00e0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-175e55d4661b9bb66d8680315cea0416, retrying]
May 03 21:25:25.099 - 148s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-193550: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-8eb304e9ec5b058bbebd5297505e00e0 expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-175e55d4661b9bb66d8680315cea0416, retrying]

... 1 lines not shown

May 03 12:05:41.838 E ns/e2e-k8s-sig-apps-job-upgrade-71 pod/foo-p67mq node/ip-10-0-245-220.us-east-2.compute.internal uid/343c61f6-adcd-4d33-87e6-5dc96a985759 container/c reason/ContainerExit code/137 cause/Error
May 03 12:07:10.670 E clusteroperator/machine-config condition/Degraded status/True reason/RequiredPoolsFailed changed: Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f5f026e04a50daf9592f5bd037ac168d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f0f3475cb57a999fdba8e296e601b1d9, retrying]
May 03 12:07:10.670 - 542s  E clusteroperator/machine-config condition/Degraded status/True reason/Unable to apply 4.12.0-0.ci-2024-05-03-102448: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-f5f026e04a50daf9592f5bd037ac168d expected 33a010772d604aff2cb625d04a9469a47f53c96e has 15d0b0288a4330b90ac89f14c781dfa7349af52c: 1 (ready 1) out of 3 nodes are updating to latest configuration rendered-master-f0f3475cb57a999fdba8e296e601b1d9, retrying]

... 1 lines not shown